STAG-VIO: Stabilized Prompt-to-Geometry Interface for Robust Dynamic Visual--Inertial Odometry

Jianyu Zhang; Jiele Zhao; Jingbin Liu; Junbin Xie; Rui Zhou; Yingze Hu

STAG-VIO: Stabilized Prompt-to-Geometry Interface for Robust Dynamic Visual--Inertial Odometry

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a track-then-segment front end that masks moving objects and discards the feature points inside them substantially improves visual-inertial odometry accuracy in highly dynamic, partially occluded scenes.

desk verdict A plausible dynamic-VIO integration of YOLOv11, adaptive SORT, and Mobile SAM that needs better empirical support before the generalization claim holds. read the letter →

arxiv 2411.19289 v4 pith:YQUL6MCE submitted 2024-11-28 cs.CV

Rui Zhou , Jingbin Liu , Junbin Xie , Jianyu Zhang , Yingze Hu , Jiele Zhao This is my paper · ORCID

classification cs.CV

keywords visual-inertialodometrydynamicenvironmentsobjectsegmentationSegmentAnythingModelSORTtrackingadaptiveKalmanfilterfeaturepointrejectionabsolutetrajectoryerror

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that visual-inertial odometry can be made substantially more accurate in dynamic scenes by running a track-then-segment front end: track every potentially moving object, cut it out with a promptable foundation model that turns a box prompt into a pixel mask, and discard any image feature points that fall inside those masks before optimization. The payoff, if the claim holds, is a single VIO pipeline that handles many kinds of moving objects, such as cars, pedestrians, and shopping carts, without retraining the segmenter for each scene. The paper reports lower absolute trajectory error than several established VIO systems on synthetic and real indoor and outdoor datasets, with the largest gains in sequences where dynamic objects dominate the view. The paper also releases a real-world visual-inertial dataset with diverse moving characters so the dynamic-object setting can be benchmarked directly.

What carries the argument

The load-bearing mechanism is the dynamic-object mask generated from the tracked boxes, because that mask decides which feature points are excluded from visual-inertial bundle adjustment. Tracking is stabilized by an adaptive Kalman update in which the measurement noise covariance is $\mathbf{R} = \mathrm{diag}(\beta\, \mathrm{erf}(\lambda\, \delta_{\mathrm{RMSE}}))$, so the filter becomes more cautious when residual errors grow. The boxes prompt Mobile SAM to segment the object at pixel level, and erosion followed by a slightly larger dilation removes mask noise while pushing the mask beyond object edges so no boundary features survive. That mask, not geometric outlier rejection, is what carries the dynamic-feature removal in the pipeline.

What would settle it

Take the released real-world dataset, hand-label every moving object in every frame, and compare the masks produced by the pipeline against those labels; if frames where the mask misses a moving object show no increase in per-frame pose error, the central claim that mask-based rejection produces the gains is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the core discovery is that a track-then-segment-then-discard pipeline plugged into a standard optimization-based VIO front end produces pose estimates that beat the compared systems in dynamic scenes. The pipeline detects candidate moving objects, associates them across frames with an adaptive Kalman filter whose measurement-noise covariance is scaled by an error function of the sliding-window residual RMSE, and uses the tracked boxes to prompt Mobile SAM, a lightweight variant of the Segment Anything foundation model, for pixel-accurate masks. KLT-tracked feature points inside the masks are removed; ORB points are re-extracted in unmasked regions under adaptive non-maximal suppression, and a compensation rule keeps the static feature count near the maximum. The paper reports lower RMSE absolute trajectory error on the VIODE high-dynamic sequences, stable tracking on OpenLORIS market scenes, and less drift than the baseline on a real-world outdoor route.

Load-bearing premise

The whole accuracy gain rests on the detector, tracker, and segmenter catching every moving object and masking exactly the dynamic pixels, with no missed moving objects and no wrongly masked static regions.

Editorial extensions

If this is right

Dynamic-object masking can be layered onto existing optimization-based VIO front ends, since the reported gains come without changing the IMU or visual bundle-adjustment core.
Sequences that previously caused VIO failure, such as the asterisked ORB-SLAM3 cases in the VIODE table, become tractable because the rejected features no longer pull the optimization.
The zero-shot segmenter should let the system move between outdoor traffic and indoor crowds without retraining, as demonstrated by the transfer from VIODE to OpenLORIS.
The SORT ablation isolates a large share of the improvement in the tracker: a 21.38% ATE reduction in the parking lot, 83.28% in city day, and 63.99% in city night.
A publicly released real-world dataset with diverse moving characters gives other methods a common benchmark for dynamic VIO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same mask-and-discard module could likely be grafted onto other VIO or SLAM systems, so the main transferable result is the pipeline rather than the specific detector, tracker, or segmenter.
Because the published numbers never measure mask precision or recall, an immediate test is to correlate per-frame mask quality against per-frame pose error on the released dataset.
A harder stress test the paper does not report is an object class the detector has never seen or an occlusion that breaks the tracker's data association; those failures would reveal how much of the universality comes from the foundation model versus the detector and tracker.
The paper lists real-time performance as future work, so the practical claim is accuracy, not yet deployment at interactive rates.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

A credible engineering integration of existing pieces into VINS-Mono: YOLOv11 detections, an enhanced SORT tracker whose Kalman measurement noise R is adaptively scaled by an erf function of residual RMSE, and Mobile SAM promptable segmentation to mask dynamic objects. The new bits are the adaptive erf-based R update and the compensation strategy based on ANMS, plus a small real-world dataset. That is a legitimate incremental contribution to dynamic VIO.

What it does well: the pipeline is described clearly enough to re-implement, the ablation shows the SORT component helps (83% ATE reduction on some VIODE sequences), and the VIODE comparison table is internally consistent. Releasing the real-world dataset is a nice gesture.

Soft spots, in order. First, the masking is motion-agnostic: every detected car, pedestrian, or cart is masked regardless of whether it moves. Static objects of those classes get their features discarded, which can only hurt the optimizer. The paper claims generalization but gives no mask precision/recall, no count of static features wrongly removed, and no check of dynamic features left in. This is the load-bearing risk, and it is not addressed. Second, the empirical support for 'outperforms state-of-the-art' is thin: no error bars or repeated trials, some OpenLORIS baselines are copied from the prior OpenLORIS paper, the real-world evaluation is qualitative only, and the key tuning parameters lambda and beta are not reported. Code is not released either. These are addressable in revision.

The stress-test concern about static-feature discard is on point; the central pose claim is not circular, and the benchmark comparison is external, so this is a matter of missing evidence rather than a fundamental flaw.

Who it is for: people working on dynamic VIO or semantic masking in visual-inertial systems. A serious referee should engage, but would ask for the missing metrics and code before accepting.

Recommendation: send to peer review, with the expectation of a major revision that adds mask-quality evaluation, repeated runs with error bars, and parameter disclosure.

Referee Report

3 major / 5 minor

Summary. The paper proposes ADUGS-VINS, a visual-inertial odometry system that combines YOLOv11 detection, an enhanced SORT tracker with an adaptive Kalman filter, and Mobile SAM promptable segmentation to identify and mask dynamic objects. Feature points falling inside the resulting masks are discarded before KLT tracking and bundle adjustment, with a compensation strategy to maintain a sufficient number of static features. The method is evaluated on the VIODE and OpenLORIS-Scene datasets and on a qualitative real-world sequence, reporting improved ATE RMSE over several published VIO baselines.

Significance. If the reported gains hold, the paper demonstrates a practical integration of a promptable foundation model into VIO for dynamic-object rejection, which is a relevant direction given the limited generalization of earlier semantic-segmentation-based approaches. The release of a real-world visual-inertial dataset is a useful contribution. However, the empirical support is currently incomplete: the key masking policy is not accompanied by any quantitative assessment of segmentation quality or false-positive rate, the OpenLORIS comparison partly relies on results from another paper, and the real-world demonstration lacks ground truth. These gaps make the central generalization claim harder to verify. The paper's core mechanism is plausible and the VIODE results are internally consistent, so the work is likely to be of interest to the VIO/dynamic-SLAM community once the evidence is strengthened.

major comments (3)

[III-A and III-B] The masking stage removes feature points for every detection of a dynamic class (vehicle, pedestrian, shopping cart, etc.) regardless of whether the object is actually moving. In scenes containing parked cars or stationary pedestrians, this policy discards useful static features and can degrade pose estimation. The paper provides no quantitative measure of mask precision/recall, no count of static features incorrectly removed, and no analysis of how many dynamic features remain in the optimizer. The ablation in Fig. 6 only compares the full method with and without the enhanced SORT, not against a motion-aware policy. Since the paper explicitly claims generalization and adaptability across environments, this motion-agnostic masking assumption is load-bearing and must be justified or mitigated.
[IV-C, Fig. 7] The OpenLORIS comparison for several baseline methods is taken directly from the prior paper [37], as acknowledged in the caption. Because the experimental protocol, parameter settings, and underlying implementations are not identical, the reported improvements may reflect differences in evaluation conditions rather than algorithmic superiority. The claim of outperforming state-of-the-art methods on OpenLORIS is therefore not fully supported. Please re-run the baselines under the same protocol as ADUGS-VINS, or restrict the 'outperforms' claim to the VIODE experiments where the comparison is self-contained.
[III-A, Eq. (4)] The adaptive measurement noise covariance model R = diag(beta * erf(lambda * delta_RMSE)) is introduced without a derivation or a discussion of its properties. In particular, erf(0)=0 is a degeneracy: if the residual RMSE is zero, the filter assumes zero measurement noise, which is unrealistic and can lead to overconfidence. The values of lambda and beta are not reported, and the ablation in Fig. 6 does not isolate the effect of this adaptive rule versus a standard SORT with fixed R. Please provide a justification for this functional form, report the parameter values, and include a sensitivity analysis or a dedicated ablation so that the contribution of the adaptive component can be assessed.

minor comments (5)

[Throughout] There are numerous typographical errors, including 'futher' (Section IV-B.3 and Conclusion), 'calulate' (Section III-A), 'their are' (Section III-A), 'ASDUGS-VINS' (Section IV-B.2), and a stray 'e' in the heading '1) ATE Comparison on VIODE Dataset: e'. A careful proofreading pass is needed.
[IV-B.2 and Fig. 6] The ablation discussion in Section IV-B.2 refers to a 'baseline' without defining it clearly, while the caption of Fig. 6 states the comparison is 'including SORT (red) and excluding SORT (blue)'. Please clarify what constitutes the baseline in the heatmap ablation and how it relates to the SORT on/off comparison.
[IV-A] The description of the OpenLORIS dataset says it contains 'visual, inertial, and odometric data,' but the experimental setup does not specify which sensor subset is used, how the VIO methods are initialized, or whether loop closure is enabled. Please provide this information, as it affects reproducibility.
[III-A] The values of the sliding-window length N, the adaptive Kalman parameters lambda and beta, and the feature parameters Nmax and Dmin used in the experiments are not reported. Please include them in the text or in a table so that the results can be reproduced.
[IV-D] The real-world experiment is only qualitative, showing trajectory overlays on satellite imagery without ground truth or quantitative error metrics. Please either provide a quantitative evaluation (e.g., with surveyed ground control points) or rephrase the claim to make clear that the real-world results are a demonstration rather than a benchmark.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pose-estimation claim is benchmarked against external datasets and no equation reduces to its own input.

full rationale

The central claim of ADUGS-VINS is that integrating an enhanced SORT tracker with Mobile SAM masks before bundle adjustment improves VIO accuracy in dynamic scenes. That claim is tested against external public benchmarks (VIODE, OpenLORIS-Scene) and a separately collected real-world dataset, so the headline result does not reduce to the paper's own fitted parameters or to a self-citation chain. The adaptive measurement-noise update in Eqs. (2)-(4) estimates R from Kalman-filter residuals; this is an online internal gain-adjustment scheme, not a quantity being predicted from the same data in a way that makes the pose result true by construction. The segmentation and tracking components are imported from external systems (YOLOv11, Mobile SAM, SORT) rather than defined in terms of the final ATE, and no uniqueness theorem or prior result by the same authors is invoked to force the method. The reader's concern that the motion-agnostic masking policy may discard static features is a genuine robustness and evaluation-gap issue, but the paper does not claim a metric that is definitionally equal to its own output. Accordingly, the derivation is self-contained with respect to circularity, even though its generalization is not fully evidenced by mask-quality statistics.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and does not introduce new physical entities. It does, however, depend on several undisclosed hyperparameters and on two strong perceptual assumptions: complete detection and segmentation of dynamic objects, and a clean separation between dynamic and static feature points. These are the main unverified premises behind the reported accuracy gains.

free parameters (5)

lambda (steepness in adaptive R update) = not reported
Controls the transition steepness of R = diag(beta * erf(lambda * delta_RMSE)) in Eq. (4); no value or selection procedure is given, and it directly affects tracking quality and hence pose accuracy.
beta (magnitude in adaptive R update) = not reported
Scales the adapted measurement noise covariance in Eq. (4); without a reported value the adaptive Kalman filter cannot be reproduced.
Nmax (maximum feature points) = not reported
Used in Section III-B for ANMS and feature compensation; the ablation heatmap in Fig. 4 sweeps over it, indicating the final operating point is a chosen value.
Dmin (minimum feature point distance) = not reported
Used in Section III-B to enforce spatial distribution of features; swept in the Fig. 4 ablation but the production value is not stated.
Residual sliding-window length N and erosion/dilation kernel sizes = not reported
Eq. (3) uses a sliding window of N frames, and Eq. (6)-(7) use structure sizes S_erode and S_dilate; none of these are specified, and all influence mask quality and filter behavior.

assumptions (4)

domain assumption The perception stack (YOLOv11 + enhanced SORT + Mobile SAM) detects and segments every dynamic object in every frame.
Section III-A builds the entire dynamic-object rejection pipeline on this assumption; missed or falsely segmented objects directly change which feature points are discarded.
domain assumption Feature points inside the segmented masks are exactly the dynamic points, and all static points lie outside the masks.
Section III-B rejects every feature point falling inside a mask; if masks leak into static regions, valid constraints are lost, and if masks undercover moving objects, dynamic features survive.
ad hoc to paper The adaptive Kalman noise model R = diag(beta * erf(lambda * delta_RMSE)) in Eq. (4) is an appropriate model of tracking measurement noise.
The erf scaling is introduced as a heuristic with no derivation or validation against alternative noise models; the claimed tracking improvements depend on this choice.
standard math The underlying VINS-Fusion optimization correctly fuses IMU and visual measurements once dynamic features are removed.
The paper builds on VINS-Fusion-style optimization; if the base VIO has structural errors, the dynamic-segmentation layer cannot repair them, though this is a reasonable background assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of STAG-VIO: Stabilized Prompt-to-Geometry Interface for Robust Dynamic Visual--Inertial Odometry." pith.science (2026). https://pith.science/paper/YQUL6MCE

@misc{pith2026241119289,
  author       = {Pith},
  title        = {Pith review of: STAG-VIO: Stabilized Prompt-to-Geometry Interface for Robust Dynamic Visual--Inertial Odometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQUL6MCE}},
  note         = {Machine review of arXiv:2411.19289}
}

read the original abstract

Dynamic visual-inertial odometry (VIO) requires reliable suppression of motion-corrupted measurements, yet prior semantic-assisted approaches depend on category-limited segmenters and degrade under partial occlusion. Promptable foundation segmentation models offer category-agnostic dynamic parsing, but their effectiveness in VIO depends critically on the temporal stability of input prompts---a factor largely overlooked in existing pipelines. When prompts derived from raw detection are jittery or intermittent under occlusion, the resulting masks flicker across frames, destabilizing geometric estimation. We propose STAG-VIO, which formulates dynamic robustness as a perception-to-geometry interface stabilization problem. We introduce uncertainty-adaptive multi-object tracking that models prompt generation as state estimation with bounded noise adaptation, producing temporally coherent box prompts. These stabilized prompts drive a lightweight foundation segmenter whose masks undergo geometry-oriented morphological refinement to establish conservative safety margins. A constraint-budget-aware feature redistribution strategy preserves well-conditioned static measurements when dynamic regions dominate the view. Experiments on VIODE and OpenLORIS-Scene show consistent gains over state-of-the-art baselines. Ablation confirms that prompt stabilization is the single most impactful component, reducing trajectory error by up to 83%.