REVIEW 4 major objections 5 minor 43 references
Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A self-supervised VO pipeline that compresses optical flow into a recurrent code and judges warped images with a discriminator reports KITTI trajectory errors about four times lower than prior self-supervised baselines, and below a full…
desk verdict A coherent self-supervised VO architecture with believable depth ablations, but the headline pose numbers rest on an undefined ATE protocol that makes the main claim unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the 'code': a 128-dimensional vector produced by an encoder that compresses optical flow between two consecutive frames. The code is carried through an LSTM, whose hidden state accumulates information from past frames, and the refined code is concatenated with the current image inside the depth network, so depth estimation is no longer single-view. Pose estimation stacks the image and estimated depth of two views into RGBD pairs. The adversarial part treats the warped target image as a generated sample and uses a discriminator, given the real target image as condition, to tell real from warped; this supplies a structural, non-pixel-wise supervision signal. A trajectory-consistency loss on eight-frame windows ties the relative poses together.
What would settle it
Run the trained model on KITTI sequences 09-10 and report ATE without any ground-truth scale alignment (or with a per-window scale fit); if the error jumps well above the reported 0.003 or above the baselines, the scale-alignment protocol rather than the model is carrying the result.
Extended reading notes
Core claim
The central claim is that treating monocular visual odometry as a sequential, adversarial image-generation task removes the main weaknesses of earlier self-supervised VO. Previous methods estimate depth from a single view and minimize pixel-wise photometric loss, which cannot detect distortion artifacts. This paper replaces single-view depth with depth conditioned on a compact code extracted from optical flow and aggregated by an LSTM, and replaces or augments the photometric loss with an adversarial discriminator that sees the synthesized frame as a whole. With these changes plus a trajectory-consistency loss, the paper reports an ATE of 0.0030±0.0014 on sequence 09 and 0.0029±0.0012 on sequence 10 of the KITTI odometry benchmark, outperforming all compared self-supervised baselines and the full ORB-SLAM baseline despite using no loop closure, bundle adjustment, or global optimization.
Load-bearing premise
The headline pose numbers assume that one globally fitted scale factor can absorb monocular scale ambiguity, so the reported trajectory error does not penalize scale drift.
Editorial extensions
If this is right
- Self-supervised monocular odometry can reach or exceed the trajectory accuracy of a classical full SLAM system on a standard driving benchmark, without loop closure or global optimization.
- Long-sequence temporal memory, not just photometric consistency, reduces scale drift and accumulated error in learned VO.
- Adversarial supervision of the warped image improves depth boundary detail, which in turn sharpens pose estimation.
- The same machinery could be extended to unsupervised end-to-end SLAM, as the paper identifies as future work.
- Because the code summarizes frame-to-frame correlation, it may serve as a reusable feature for other video-based 3D tasks.
Reading between the lines
- A natural test the paper does not run is to report ATE without the single scale-factor alignment to ground truth; that would separate genuine scale-consistent shape accuracy from monocular scale drift, which the current metric absorbs.
- The LSTM window is fixed at 15 frames; an ablation varying its length would reveal whether accuracy saturates quickly or keeps rising with longer memory.
- The discriminator's contribution could be probed by replacing the adversarial loss with another structural loss, such as deeper feature-matching penalties, to isolate what GAN training adds beyond the code and LSTM.
- The compact code could be evaluated for transfer, e.g., training on one dataset and testing in unseen environments, to see whether the frame-to-frame correlation representation generalizes better than raw-image methods.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a self-supervised monocular visual odometry framework that combines a compact optical-flow-derived code, LSTM-based temporal aggregation, depth and pose networks, a trajectory consistency loss, and a GAN-based adversarial loss on the warped target image. Depth is estimated from the current image conditioned on the refined code, and pose is regressed from RGBD inputs. Training uses photometric/SSIM appearance loss, edge-aware smoothness, trajectory consistency, and a discriminator. Experiments on KITTI and Cityscapes report depth accuracy comparable to or better than self-supervised baselines, and pose ATE on KITTI sequences 09-10 claimed to significantly outperform both self-supervised methods and ORB-SLAM full. Ablations show monotonic improvements in pose and depth as the code, GAN, LSTM, and trajectory consistency components are added.
Significance. If the pose results are reproducible under a well-defined evaluation protocol, the paper would demonstrate a strong self-supervised VO result: a recurrent adversarial framework beating full ORB-SLAM on KITTI 09-10 while using only self-supervision. The depth gains are modest but plausible, and the ablation story is internally consistent, with each added component contributing a small improvement. I found no circular use of ground-truth pose or depth in training; the supervision comes from photometric, SSIM, smoothness, trajectory-consistency, and adversarial losses. The main weakness is that the headline pose claim currently rests on an underspecified ATE protocol and a test-time scale fit that may not measure the scale-drift reduction claimed as a contribution. Because no code or predicted trajectories are provided, the central result is not verifiable as written.
major comments (4)
- [Section 4.3, Table 2] The ATE evaluation protocol is not defined. The text says ATE is evaluated for both translation and rotation, but Table 2 reports a single number per method and does not state the alignment procedure (single global scale factor, Umeyama/Sim(3), or start-frame alignment), the units, or how the baseline numbers were obtained. Because this table is the only evidence for the central claim that the method significantly outperforms all other baselines, the authors must specify the metric exactly and report per-sequence rotation errors and predicted trajectories.
- [Section 4.3, Eq. (14), contributions] The test-time scaling statement is load-bearing. The paper states that a scaling factor is used to align the trajectories with ground truth [40], while also claiming as a contribution that the method reduces scale drift. A single global scale factor can absorb a constant scale error but not time-varying scale drift; if the reported ATE is computed after this fit, it does not penalize scale drift and the claimed advantage may vanish under a scale-drift-sensitive metric. Please report ATE with and without global scale alignment and quantify scale drift separately, for example through trajectory-length error or per-segment scale error.
- [Section 4.3, Table 2] The magnitude of the reported pose numbers is surprising and needs concrete grounding. If the ATE entries are in meters, values near 0.003 m on sequences 09 and 10 are below the typical accuracy of the KITTI ground-truth poses (RTK GPS/IMU, centimeter-level), so the reader cannot tell whether the numbers are normalized by trajectory length, measured in different units, or computed on a subset. Please state the exact normalization and verify the result by reporting raw translation error before alignment, trajectory length, and the alignment residual.
- [Section 3.5, Eq. (9)] The GAN objective is written with D(It|It) for the real term, conditioning the discriminator on the very image it judges. For a conditional image-generation formulation of view synthesis, the condition should be the source image or the two input frames, not the target image; as written, the real term is trivially distinguishable. Please correct the notation or clarify the implementation, since the adversarial loss is one of the two main contributions of the paper.
minor comments (5)
- [Section 4.3] The text contains typos: 'Vid2Dpeth' should be 'Vid2Depth', and 'emplemented' should be 'implemented'.
- [Table 1 caption] The caption reads 'Monodular depth estimation'; this should be 'Monocular depth estimation'.
- [Sections 3.6 and 5] There are grammar errors such as 'an self-supervised' in the conclusions and 'suffice' in Section 3.6; these should be corrected.
- [Section 3.1] The dependence on Farneback optical flow as the input representation should be stated explicitly as an implementation detail, since at test time the method requires an external optical-flow computation rather than being fully end-to-end from raw images.
- [Availability] No code or predicted trajectories are provided; given the central pose claim, releasing either would greatly aid verification of the reported results.
Circularity Check
No significant circularity: the pose/depth predictions are produced by a self-supervised network and the only ground-truth alignment is a standard test-time evaluation nuisance parameter.
full rationale
The paper's derivation chain is self-contained and does not reduce to its inputs. The training losses in Sec. 3.6 (photometric Eq. 7, SSIM Eq. 10, smoothness Eq. 11, trajectory consistency Eq. 13, and GAN Eq. 9) are all computed from unlabeled image sequences and network outputs; no ground-truth pose or depth is injected during training. The trajectory consistency loss enforces that concatenated relative poses equal the directly estimated pose over the same interval, which is a self-consistency constraint rather than a circular definition of accuracy. The pose and depth networks are trained jointly and evaluated on held-out KITTI sequences 09-10, so the Table 2 ATE values are genuine outputs rather than fitted parameters. The two test-time scaling operations (depth median scaling in Sec. 4.2 and 'a scaling factor is used to align the trajectories with ground truth [40]' in Sec. 4.3) are standard monocular evaluation protocol: they remove the global scale ambiguity by fitting a single nuisance parameter, after which the remaining ATE still measures trajectory shape error. This does not make the prediction equivalent to its input by construction, though it may affect how strongly the results support the paper's scale-drift claim. The self-citations [36, 37] appear only in the related-work discussion and are not load-bearing for any derivation. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result was found. The reported ATE protocol is underspecified in the paper, but that is an evaluation-transparency and correctness concern, not circularity.
Assumptions & free parameters
free parameters (6)
- test-time pose scale alignment factor =
not reported; fit per test sequence to ground truth trajectory
- test-time depth median scaling factor =
not reported; computed per test image to match ground-truth median
- loss weights lambda_a, lambda_s, lambda_t, lambda_g =
0.75, 0.1, 0.14, 0.01
- appearance loss weight alpha =
0.85
- code dimension =
128
- LSTM sequence length =
15
assumptions (5)
- domain assumption View synthesis assumes a static scene with no illumination change; violations are handled by a per-pixel mask.
- domain assumption Farneback optical flow provides usable frame-to-frame correspondence as input to the encoder.
- domain assumption The LSTM trained on length-15 sequences generalizes to full test trajectories processed sequentially.
- domain assumption The generative adversarial setup with WGAN converges to a useful equilibrium and the discriminator's judgments transfer to sharper depth.
- standard math The rigid-body transformation composition relations in Eq. (12) are applied correctly to compare direct and concatenated poses.
Cite this review
Pith. "Pith review of Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry." pith.science (2026). https://pith.science/paper/MW3LFI55
@misc{pith2026190808704,
author = {Pith},
title = {Pith review of: Sequential Adversarial Learning for Self-Supervised Deep Visual Odometry},
year = {2026},
howpublished = {\url{https://pith.science/paper/MW3LFI55}},
note = {Machine review of arXiv:1908.08704}
}
read the original abstract
We propose a self-supervised learning framework for visual odometry (VO) that incorporates correlation of consecutive frames and takes advantage of adversarial learning. Previous methods tackle self-supervised VO as a local structure from motion (SfM) problem that recovers depth from single image and relative poses from image pairs by minimizing photometric loss between warped and captured images. As single-view depth estimation is an ill-posed problem, and photometric loss is incapable of discriminating distortion artifacts of warped images, the estimated depth is vague and pose is inaccurate. In contrast to previous methods, our framework learns a compact representation of frame-to-frame correlation, which is updated by incorporating sequential information. The updated representation is used for depth estimation. Besides, we tackle VO as a self-supervised image generation task and take advantage of Generative Adversarial Networks (GAN). The generator learns to estimate depth and pose to generate a warped target image. The discriminator evaluates the quality of generated image with high-level structural perception that overcomes the problem of pixel-wise loss in previous methods. Experiments on KITTI and Cityscapes datasets show that our method obtains more accurate depth with details preserved and predicted pose outperforms state-of-the-art self-supervised methods significantly.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[40]
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised Learn- ing of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction. In CVPR, 2018
work page 2018
-
[1]
Generative Adversarial Networks for Unsupervised Monocular Depth Prediction
Filippo Aleotti, Fabio Tosi, Matteo Poggi, and Stefano Mat- toccia. Generative Adversarial Networks for Unsupervised Monocular Depth Prediction. In ECCV, 2018
work page 2018
-
[2]
Wasserstein Generative Adversarial Networks
Martin Arjovsky, Soumith Chintala, and L ´eon Bottou. Wasserstein Generative Adversarial Networks. In ICML, 2017
work page 2017
-
[3]
A Deeper Insight into the UnDEMoN: Unsupervised Deep Network for Depth and Ego-Motion Estimation
V Babu, Anima Majumder, Kaushik Das, Swagat Kumar, et al. A Deeper Insight into the UnDEMoN: Unsupervised Deep Network for Depth and Ego-Motion Estimation. arXiv preprint arXiv:1809.00969, 2018
work page Pith review arXiv 2018
-
[4]
UnDEMoN: Unsupervised Deep Network for Depth and Ego-Motion Estimation
V Madhu Babu, Kaushik Das, Anima Majumdar, and Swagat Kumar. UnDEMoN: Unsupervised Deep Network for Depth and Ego-Motion Estimation. In IROS, 2018
work page 2018
-
[5]
Dan Barnes, Will Maddern, Geoffrey Pascoe, and Ingmar Posner. Driven to Distraction: Self-Supervised Distractor Learning for Robust Monocular Visual Odometry in Urban Environments. In ICRA, 2018
work page 2018
-
[6]
CodeSLAM: Learn- ing a Compact, Optimisable Representation for Dense Visual SLAM
Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davison. CodeSLAM: Learn- ing a Compact, Optimisable Representation for Dense Visual SLAM. In CVPR, 2018
work page 2018
-
[7]
DeepDriving: Learning Affordance for Direct Percep- tion in Autonomous Driving
Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. DeepDriving: Learning Affordance for Direct Percep- tion in Autonomous Driving. In ICCV, 2015
work page 2015
Show all 43 references
-
[8]
The Cityscapes Dataset for Semantic Urban Scene Understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In CVPR, 2016
2016
-
[9]
FlowNet: Learning Optical Flow with Convolutional Networks
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow with Convolutional Networks. In ICCV, 2015
2015
-
[10]
Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network
David Eigen, Christian Puhrsch, and Rob Fergus. Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network. In NIPS, 2014
2014
-
[11]
Direct Sparse Odometry
Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct Sparse Odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):611–625, 2018
2018
-
[12]
LSD- SLAM: Large-Scale Direct Monocular SLAM
Jakob Engel, Thomas Sch ¨ops, and Daniel Cremers. LSD- SLAM: Large-Scale Direct Monocular SLAM. In ECCV, 2014
2014
-
[13]
Two-Frame Motion Estimation Based on Polynomial Expansion
Gunnar Farnebck. Two-Frame Motion Estimation Based on Polynomial Expansion. In Scandinavian Conference on Im- age Analysis, 2003
2003
-
[14]
Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles
Christian Forster, Simon Lynen, Laurent Kneip, and Davide Scaramuzza. Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles. In IROS, 2013
2013
-
[15]
SVO: Fast Semi-Direct Monocular Visual Odometry
Christian Forster, Matia Pizzoli, and Davide Scaramuzza. SVO: Fast Semi-Direct Monocular Visual Odometry. In ICRA, 2014
2014
-
[16]
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue. In ECCV, 2016
2016
-
[17]
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Cl ´ement Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised Monocular Depth Estimation with Left-Right Consistency. In CVPR, 2017
2017
-
[18]
Generative Adversarial Nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. InNIPS, 2014
2014
-
[19]
MapNet: An Allocen- tric Spatial Memory for Mapping Environments
Joao F Henriques and Andrea Vedaldi. MapNet: An Allocen- tric Spatial Memory for Mapping Environments. In CVPR, 2018
2018
-
[20]
Long Short-Term Memory
Sepp Hochreiter and J ¨urgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 1997
1997
-
[21]
Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, Junyan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR, 2017
2017
-
[22]
Geometric Consistency for Self- Supervised End-to-End Visual Odometry
Ganesh Iyer, J Krishna Murthy, Gunshi Gupta, Madhava Krishna, and Liam Paull. Geometric Consistency for Self- Supervised End-to-End Visual Odometry. In CVPR Work- shops, 2018
2018
-
[23]
Dense visual SLAM for RGB-D cameras
Christian Kerl, Jurgen Sturm, and Daniel Cremers. Dense visual SLAM for RGB-D cameras. In IROS, 2014
2014
-
[24]
Adam: A method for Stochastic Optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for Stochastic Optimization. In ICLR, 2015
2015
-
[25]
Parallel Tracking and Map- ping on a Camera Phone
Georg Klein and David Murray. Parallel Tracking and Map- ping on a Camera Phone. In ISMAR, 2009
2009
-
[26]
UndeepVO: Monocular Visual Odometry through Unsuper- vised Deep Learning
Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu. UndeepVO: Monocular Visual Odometry through Unsuper- vised Deep Learning. In ICRA, 2018
2018
-
[27]
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields. IEEE Transactions on Pat- tern Analysis and Machine Intelligence , 38(10):2024–2039, 2016
2024
-
[28]
Un- supervised Learning of Depth and Ego-Motion from Monoc- ular Video Using 3D Geometric Constraints
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Un- supervised Learning of Depth and Ego-Motion from Monoc- ular Video Using 3D Geometric Constraints. InCVPR, 2018
2018
-
[29]
ORB-SLAM: A Versatile and Accurate Monoc- ular SLAM System
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. ORB-SLAM: A Versatile and Accurate Monoc- ular SLAM System. IEEE Transactions on Robotics , 31(5):1147–1163, 2015
2015
-
[30]
KinectFusion: Real-Time Dense Surface Mapping and Tracking
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgib- bon. KinectFusion: Real-Time Dense Surface Mapping and Tracking. In ISMAR, 2011
2011
-
[31]
Colored Point Cloud Registration Revisited
Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Colored Point Cloud Registration Revisited. In ICCV, 2017
2017
-
[32]
Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan. PyTorch. https://github.com/pytorch/ pytorch, 2017
2017
-
[33]
DeMoN: Depth and Motion Network for Learning Monocular Stereo
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Niko- laus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. DeMoN: Depth and Motion Network for Learning Monocular Stereo. In CVPR, 2017
2017
-
[34]
DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks. In ICRA, 2017
2017
-
[35]
Image Quality Assessment: from Error Visibility to Structural Similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simon- celli, et al. Image Quality Assessment: from Error Visibility to Structural Similarity. IEEE Transactions on Image Pro- cessing, 13(4):600–612, 2004
2004
-
[36]
Guided Feature Selection for Deep Visual Odometry
Fei Xue, Qiuyuan Wang, Xin Wang, Wei Dong, Junqiu Wang, and Hongbin Zha. Guided Feature Selection for Deep Visual Odometry. In ACCV, 2018
2018
-
[37]
Beyond Tracking: Selecting Mem- ory and Refining Poses for Deep Visual Odometry
Fei Xue, Xin Wang, Shunkai Li, Qiuyuan Wang, Junqiu Wang, and Hongbin Zha. Beyond Tracking: Selecting Mem- ory and Refining Poses for Deep Visual Odometry. InCVPR, 2019
2019
-
[38]
Deep Virtual Stereo Odometry: Leveraging Deep Depth Pre- diction for Monocular Direct Sparse Odometry
Nan Yang, Rui Wang, Jorg Stuckler, and Daniel Cremers. Deep Virtual Stereo Odometry: Leveraging Deep Depth Pre- diction for Monocular Direct Sparse Odometry. In ECCV, 2018
2018
-
[39]
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
Zhichao Yin and Jianping Shi. GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose. In CVPR, 2018
2018
-
[41]
DeepTAM: Deep Tracking and Mapping
Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox. DeepTAM: Deep Tracking and Mapping. In ECCV, 2018
2018
-
[42]
Unsupervised Learning of Depth and Ego-Motion from Video
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised Learning of Depth and Ego-Motion from Video. In CVPR, 2017
2017
-
[43]
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired Image-to-Image Translation Using Cycle- Consistent Adversarial Networks. In ICCV, 2017
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.