Pith. sign in

REVIEW 5 major objections 3 minor 14 references

Good Deep Features to Track: Self-Supervised Feature Extraction and Tracking in Visual Odometry

T0 review · 5 major / 3 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Self-supervised fine-tuning with a VO's own reprojection errors makes deep features spread more evenly across rigid structures and improves estimated trajectories in a maritime dataset.

desk verdict A plausible self-supervised fine-tuning idea for feature extractors in VO, honestly presented as a proof of concept, but the evidence is qualitative and the labeling loop is genuinely circular. read the letter →

arxiv 2509.08333 v1 pith:4OXYXCZN submitted 2025-09-10 cs.RO cs.CV

Sai Puneeth Reddy Gottam , Haoming Zhang , Eivydas Keras This is my paper · ORCID
classification cs.ROcs.CV
keywords self-supervisedlearningvisualodometryfeatureextractiontrackingSuperPointGluemaritimerobotnavigationRTAB-Map
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that deep feature extractors like SuperPoint, though better than handcrafted detectors at spreading features, still misbehave on out-of-distribution outdoor scenes: they waste detections on water, terrain, and other unreliable regions. The authors propose a self-supervised loop in which a visual odometry system (RTAB-Map with SuperPoint and SuperGlue) labels its own reprojection-consistent features as 'good', and these labels fine-tune SuperPoint through a weighted loss on score maps, peakiness, and descriptors. Preliminary results on the Pohang maritime dataset show the fine-tuned model suppresses water and terrain features, spreads detections across rigid structures, and traces qualitatively better vessel trajectories at daytime and nighttime. The contribution is a proof of concept that task-level feedback from a VO can re-shape a generic feature extractor without manually labelled keypoints.

What carries the argument

The mechanism is a self-supervised training loop over the feature extractor, with a total weighted loss L = w_i·L_i(X,Y) + w'_i·L_i(X',Y') + w_pk·L_pk + w_d·L_d(D,D',S). The labels Y and correspondences come from the VO's own stereo and temporal matching and its estimated relative pose; reprojection errors separate tracked 'good' features from unreliable ones. The peaky loss L_pk penalizes diffuse or ambiguous keypoint score maps (the detector's per-pixel confidence), and the descriptor hinge loss L_d enforces separation of non-matching descriptors. This turns the VO into a label generator for the feature extractor, so the extractor is optimized for the actual geometry of the task rather tha

What would settle it

Fix the RTAB-Map pipeline and compare trajectory error on a held-out sequence (e.g., a new Pohang run or KITTI sequence not used for qualitative checks) between the pre-trained and fine-tuned SuperPoint. If the fine-tuned model does not reduce mean trajectory error, or if it concentrates features on rigid structures but worsens pose estimation in textureless offshore scenes, the central claim fails. A quicker probe: add calibrated noise to the relative pose used to label features during training; if the fine-tuned features still spread evenly and still improve trajectories, then reprojection-e

Watch

Extended reading notes

Core claim

The central claim is that 'good deep features to track' can be defined by the downstream task itself: a feature is good if, given the VO's estimated relative pose, it yields low reprojection error and is consistently tracked across stereo and temporal frames. Using this definition, the paper fine-tunes SuperPoint inside a self-supervised pipeline. The updated extractor produces score maps that concentrate on rigid infrastructure and suppress terrain and water surfaces, and the resulting features, fed back into RTAB-Map, give reconstructed vessel trajectories that stay closer to the expected paths in the Pohang dataset than trajectories from pre-trained SuperPoint or ORB features. The paper i

Load-bearing premise

The training labels for 'good' features are produced by the VO's own estimated relative pose and reprojection errors, so the method assumes those pose estimates are accurate enough in the very scenes—low-texture, dynamic, harsh lighting—where the extractor is supposed to improve.

Editorial extensions

If this is right

  • Fine-tuning with task feedback generalizes better than static pre-trained features in environments that look unlike the training data, especially low-texture maritime scenes.
  • The same label-generation loop can be applied to other VO/VSLAM pipelines and other feature extractors, turning any pose estimator into a supervisor for its own front-end.
  • Features become more evenly distributed over rigid structures, giving more stable geometric constraints for rotation from distant points and translation from nearby points.
  • Trajectory improvement appears in both daytime and nighttime narrow-canal sequences; in offshore areas the benefit is less pronounced, suggesting the method helps most when texture is scarce but structures exist.
  • The proof of concept implies label-intensive feature annotation can be replaced by geometric consistency labels, reducing the cost of adapting detectors to new domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same self-supervision could be iterated: after fine-tuning the extractor, the improved VO poses produce cleaner labels, so multiple rounds might converge to a fixed point where extractor and pose estimator are mutually consistent; a natural extension is to monitor whether such iteration stabilizes or drifts.
  • Since SuperGlue was left frozen, the loss's descriptor term had no effect on matching; fine-tuning the matcher jointly might amplify the gains, or reveal that descriptor updates are unnecessary when the extractor alone reshapes the score maps.
  • A quantitative coverage metric (e.g., number of features on water vs. structure, or entropy of feature distribution) could turn the qualitative 'more equitably spread' observation into a testable objective; the authors don't provide one, but the claim implies a measurable redistribution.
  • The nighttime offshore result—SuperPoint drifts less at night than day—suggests the model implicitly learns to ignore reflections and moving objects; one could probe this by testing on scenes with controlled specularities or dynamic boats.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The manuscript proposes a self-supervised fine-tuning procedure for the SuperPoint feature extractor within an RTAB-Map visual odometry pipeline, with SuperGlue as the matcher. The key idea is to use the VO's own estimated relative pose and the resulting reprojection errors to label image-region-specific 'good' features, and then train SuperPoint with a weighted loss (Eq. 1) that combines keypoint losses on original and warped images, a peaky loss, and a descriptor hinge loss. The authors present qualitative feature maps on Pohang and KITTI data and trajectory plots on Pohang sequences, claiming that the self-supervised model produces more uniformly distributed features on rigid structures while suppressing unreliable regions such as water and terrain, and that this improves trajectory estimation in narrow and offshore scenes. The paper explicitly states that the work is at an early stage, that SuperGlue was not retrained, and that the current results are 'not yet sufficient' to fully answer what defines good deep features to track.

Significance. If validated, the proposed self-supervised adaptation is practically valuable: it could adapt generic deep feature extractors to a target deployment environment without manual annotation, using only geometric consistency signals derived from an existing VO/SLAM system. The idea of task-specific feedback for feature learning is interesting, and the use of a real maritime dataset (Pohang) strengthens the motivation. The paper also has the virtue of making a concrete falsifiable claim: fine-tuning SuperPoint with the proposed loss changes feature distribution and improves VO trajectory quality. However, the current evidence is almost entirely qualitative, and the self-supervision signal is derived from the very VO system the method is meant to improve, creating a circularity risk that is not addressed experimentally. The absence of quantitative metrics, training details, and a controlled study of the label-generation mechanism means the central claim is not yet established.

major comments (5)
  1. [Section III.B, Fig. 5] The central claim that the self-supervised SuperPoint 'outperforms' the other VO variants is not supported by any quantitative evidence. The trajectory plots show no ground truth, and no ATE, RPE, scale-normalized error, or statistics over repeated runs are reported. Without such metrics, the reader cannot assess which trajectory is more accurate, especially since differences appear mainly as drift direction and shape. Please add quantitative error measures on Pohang (and, if possible, on KITTI sequence 06 with known ground truth), including per-sequence numbers and, ideally, error bars or multiple runs. An ablation that isolates the effect of the proposed loss from random fine-tuning would also be needed.
  2. [Section II, Eq. (1)] The labels for 'good' features are computed from reprojection errors that use the relative pose estimated by RTAB-Map from the same SuperPoint features being trained. This is a potential circular dependency: if the VO pose estimate is biased in the low-texture or dynamic scenes the method targets, features that are consistently wrong under the biased pose can still produce low reprojection error and will be labeled 'good'. The paper provides no evidence that the RTAB-Map poses on Pohang are accurate enough for the labels to be trustworthy. A concrete test is needed: e.g., compare the labels against ground-truth pose on a dataset with known poses, or use an independent pose estimator to generate labels, or demonstrate that the label quality improves as training proceeds. Without this, the training signal may reinforce the original feature extractor's biases rather than correct them.
  3. [Section II, Eq. (1)] The loss is defined only symbolically. The weights w_i, w_i', w_pk, w_d, the reprojection error threshold, and the exact procedure for constructing Y and Y' are not specified. The 'peaky loss' and the descriptor hinge loss L_d are described in one sentence each, which is insufficient to reproduce or analyze the method. Training details (learning rate, number of iterations, dataset split, warm-start, whether both stereo and temporal correspondences are used in each batch) are also omitted. Without these, the reported qualitative results cannot be reproduced or compared against alternative self-supervised schemes.
  4. [Section III.A, Figs. 3-4] The claim that features become 'more equitably spread' and that unreliable regions (terrain, water) are suppressed is made only through qualitative feature maps. There is no quantitative measure of feature distribution (e.g., coverage, cell occupancy, nearest-neighbour spacing), repeatability, track length, or the fraction of features that fall on the water/terrain. Moreover, Fig. 4 is captioned as KITTI seq. 06 while the text in Section III.B says 'see Fig. 4' when discussing Pohang; please clarify which dataset each panel shows and add quantitative distribution statistics for both Pohang and KITTI.
  5. [Section IV] The authors explicitly state that they did not train the feature matcher SuperGlue, which remains pre-trained on the original SuperPoint, and that 'current results are also not yet sufficient' to answer the paper's central question. Since the title and abstract state that the method improves 'feature extraction and tracking', and the pipeline includes SuperGlue, this is a load-bearing limitation. The claims should either be explicitly restricted to feature extraction, or SuperGlue should be jointly fine-tuned and the trajectory results re-evaluated. As written, the tracking component is not actually modified.
minor comments (3)
  1. [Section II, Eq. (1)] The notation X, X', Y, Y', D, D', S is used without formal definitions. Please define the score map, descriptor map, label map, and the warping operation precisely, including dimensions.
  2. [Fig. 5] The legend includes a 'Baseline' trajectory that is never defined in the text. Also, the axis labels and scales are inconsistent across the four subplots, making visual comparison difficult. It would help to plot all methods on the same coordinate scale with a common origin.
  3. [General] There are several cross-reference inconsistencies (e.g., 'see Fig. 4' when discussing Fig. 5; Fig. 3 vs Fig. 4 refer to different datasets in different paragraphs). Please correct these and ensure every figure is cited at the point where it is discussed.

Circularity Check

1 steps flagged · score 6.0 of 10

Training labels come from the VO's own pose estimate, making the reported feature improvements a restatement of the loss objective.

  1. self definitional [Section II (Eq. 1) and Section III-A]
    "During training, estimated relative pose, stereo and temporal correspondences are used to evaluate reprojection errors, which serve as geometric consistency cues. Features that are well-tracked and geometrically reliable are identified as “good” and used to compute a total weighted loss... Compared to the former, features extracted by the self-supervised model tend to be more equitably spread across expected regions in the image, such as rigid structures, while suppressing features from less reliable areas like terrain and water surfaces."

    The 'good' feature labels are computed from the relative pose estimated by RTAB-Map, which uses the same SuperPoint/SuperGlue features being trained. The loss therefore rewards exactly the features that are internally consistent with the VO's current pose. The reported 'improvement' — suppression of terrain/water and more uniform coverage — is the loss objective restated as a result, not an independent finding. The same feedback loop means a biased pose (e.g., tracking water reflections) would label those biased features as good, and fine-tuning would amplify that bias. Trajectory comparisons lack ground truth, so there is no external check on whether the trained features are actually better.

full rationale

The paper's claimed derivation is an empirical training-evaluation loop rather than a formal derivation. The central supervision signal is geometric consistency: features are labeled 'good' if they have low reprojection error under the pose estimated by RTAB-Map. Because RTAB-Map's pose is itself computed from the features being trained, the label is a function of the model's current predictions. This means the observed post-training behavior (suppression of terrain/water, more even distribution) is not independent evidence that the features are better; it is the direct result of optimizing the loss that defines 'good' in terms of those same reprojection errors. The trajectory plots contain no ground truth, so the reader cannot independently verify that suppressing these regions actually improves localization. The paper's own caveat ('current results are also not yet sufficient') supports this. No self-citations are load-bearing, and the method is described transparently, but the central validation suffers from a self-confirming loop.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on several hand-chosen loss weights and a reprojection threshold, all unreported. The central assumption is that VO pose estimates are reliable enough to generate correct labels in the very scenes where the method targets improvement. No new entities are introduced.

free parameters (5)
  • w_i (label loss weight) = not reported
    Weight for the supervised keypoint loss in Eq. (1), chosen by hand without reported values.
  • w_i' (warped image loss weight) = not reported
    Weight for the warped-image keypoint loss in Eq. (1), not specified.
  • w_pk (peaky loss weight) = not reported
    Weight penalizing diffuse score maps, chosen ad hoc.
  • w_d (descriptor loss weight) = not reported
    Weight for the descriptor hinge loss, chosen ad hoc.
  • reprojection error threshold = not reported
    Threshold that defines 'good' vs 'poor' features when generating self-supervised labels from VO reprojection errors.
assumptions (4)
  • domain assumption Pre-trained SuperPoint and SuperGlue are valid starting points.
    The method assumes the off-the-shelf models already capture useful features so that fine-tuning is sufficient.
  • domain assumption RTAB-Map is a representative VO/SLAM baseline.
    The paper uses RTAB-Map exclusively and generalizes conclusions about VO performance from it.
  • domain assumption The VO's own relative pose estimates are accurate enough to label 'good' features.
    Section II uses these estimates to compute reprojection errors; if pose is wrong in the target scenes, the labels are wrong.
  • ad hoc to paper The total weighted loss in Eq. (1) is an appropriate objective.
    The loss terms are proposed by the authors without ablation or justification beyond intuition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Good Deep Features to Track: Self-Supervised Feature Extraction and Tracking in Visual Odometry." pith.science (2026). https://pith.science/paper/4OXYXCZN

@misc{pith2026250908333,
  author       = {Pith},
  title        = {Pith review of: Good Deep Features to Track: Self-Supervised Feature Extraction and Tracking in Visual Odometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OXYXCZN}},
  note         = {Machine review of arXiv:2509.08333}
}
read the original abstract

Visual-based localization has made significant progress, yet its performance often drops in large-scale, outdoor, and long-term settings due to factors like lighting changes, dynamic scenes, and low-texture areas. These challenges degrade feature extraction and tracking, which are critical for accurate motion estimation. While learning-based methods such as SuperPoint and SuperGlue show improved feature coverage and robustness, they still face generalization issues with out-of-distribution data. We address this by enhancing deep feature extraction and tracking through self-supervised learning with task specific feedback. Our method promotes stable and informative features, improving generalization and reliability in challenging environments.

Figures

Figures reproduced from arXiv: 2509.08333 by the authors.

Figure 1
Figure 1. Extracted features (yellow) of visual odometry with different state-of-the-art approaches using the Pohang dataset (Pohang 04) [5]. arXiv:2509.08333v1 [cs.RO] 10 Sep 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Proposed self-supervised pipeline to train the SuperPoint [2] and SuperGlue [9] models within a well-established visual SLAM framework RTAB-Map [11]. Although many methods exist to improve feature quantity — such as sophisticated feature matching and filtering [1], optimizing camera attributes (e.g., exposure time) [6], and removing dynamic objects [7] using semantic information [8] — they still struggle to generali… view at source ↗
Figure 3
Figure 3. Extracted visual features (purple and green) of Pohang data [5] in different scenes and datetime using original pre-trained (top row) and self-supervised (bottom row) SuperPoint model. (a) Features from pre-trained SuperPoint. (b) Features from self-supervised SuperPoint [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Extracted visual features (red) of KITTI data (seq. 06) [12]. We evaluate all methods on the Pohang dataset [5], which is divided into sequences recorded in narrow and wide water canals, during both daytime and nighttime (see [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Estimated trajectories of both narrow canal and offshore areas in Pohang [5] using different feature extractors implemented in RTAB-Map [11] [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: Demonstration of robot localization using gnssFGO [14]. The green, white, and blue trajectories represent the baseline, GNSS standalone, and gnssFGO estimates. ACKNOWLEDGMENT The presented research was supported by the German Federal Ministry of Economic Affairs and Cl…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

  1. [1]

    SOFT2: Stereo visual odom- etry for road vehicles based on a point-to-epipolar-line metric,

    I. Cvi ˇsi´c, I. Markovi ´c, and I. Petrovi ´c, “SOFT2: Stereo visual odom- etry for road vehicles based on a point-to-epipolar-line metric,”IEEE Transactions on Robotics, vol. 39, no. 1, pp. 273–288, 2023

    work page 2023

  2. [2]

    SuperPoint: Self- supervised interest point detection and description,

    D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperPoint: Self- supervised interest point detection and description,” in2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 337–33 712

    work page 2018

  3. [3]

    Distinctive image features from scale-invariant keypoints,

    D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vision, vol. 60, no. 2, pp. 91–110, Nov. 2004

    work page 2004

  4. [4]

    Good features to track,

    J. Shi and Tomasi, “Good features to track,” in1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1994, pp. 593–600

    1994

  5. [5]

    Pohang canal dataset: A multimodal maritime dataset for autonomous navigation in restricted waters,

    D. Chung, J. Kim, C. Lee, and J. Kim, “Pohang canal dataset: A multimodal maritime dataset for autonomous navigation in restricted waters,”The International Journal of Robotics Research, vol. 42, no. 12, pp. 1104–1114, 2023

    work page 2023

  6. [6]

    Proactive camera attribute control using Bayesian optimization for illumination-resilient visual navigation,

    J. Kim, Y . Cho, and A. Kim, “Proactive camera attribute control using Bayesian optimization for illumination-resilient visual navigation,”IEEE Transactions on Robotics, vol. 36, no. 4, pp. 1256–1271, 2020

    work page 2020

  7. [7]

    Moving on to dynamic environ- ments: Visual odometry using feature classification,

    B. Kitt, F. Moosmann, and C. Stiller, “Moving on to dynamic environ- ments: Visual odometry using feature classification,” in2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2010, pp. 5551–5556

    work page 2010

  8. [8]

    Dynamic SLAM: A visual SLAM in outdoor dynamic scenes,

    S. Wen, X. Li, X. Liu, J. Li, S. Tao, Y . Long, and T. Qiu, “Dynamic SLAM: A visual SLAM in outdoor dynamic scenes,”IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1–11, 2023

    work page 2023

Show all 14 references
  1. [9]

    Super- Glue: Learning feature matching with graph neural networks,

    P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Super- Glue: Learning feature matching with graph neural networks,” in2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 4937–4946

    2020

  2. [10]

    Rein- forcement learning meets visual odometry,

    N. Messikommer, G. Cioffi, M. Gehrig, and D. Scaramuzza, “Rein- forcement learning meets visual odometry,” inComputer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LIX. Berlin, Heidelberg: Springer-Verlag, 2024, p. 76–92

    2024

  3. [11]

    RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation,

    M. Labb ´e and F. Michaud, “RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation,”Journal of Field Robotics, vol. 36, no. 2, pp. 416–446, 2019

    2019

  4. [12]

    Are we ready for autonomous driving? the KITTI vision benchmark suite,

    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the KITTI vision benchmark suite,” in2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354–3361

    2012

  5. [13]

    ORB: An efficient alternative to sift or surf,

    E. Rublee, V . Rabaud, K. Konolige, and G. Bradski, “ORB: An efficient alternative to sift or surf,” in2011 International Conference on Computer Vision, 2011, pp. 2564–2571

    2011

  6. [14]

    GNSS/Multisen- sor fusion using continuous-time factor graph optimization for robust localization,

    H. Zhang, C.-C. Chen, H. Vallery, and T. D. Barfoot, “GNSS/Multisen- sor fusion using continuous-time factor graph optimization for robust localization,”IEEE Transactions on Robotics, vol. 40, pp. 4003–4023, 2024

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.