REVIEW 3 major objections 5 minor 42 references
Adversarial View-Consistent Learning for Monocular Depth Estimation
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Monocular depth networks trained with an adversarial pose generator that warps predictions into hard views produce view-consistent depth maps and outperform their single-view baselines on NYU Depth v2.
desk verdict A plausible view-consistency training trick with a geometrically sound core, undermined by inconsistent tables that need resolving before the empirical claims can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the differentiable depth-map warping operation $W(D_s, p_{s→t})$, which projects each source pixel into a target camera frame using the camera intrinsics $K$ and a 6-DoF pose, then fills each target pixel with the minimum depth among all source points that land on it, treating empty pixels as ignored. This z-buffer rule keeps the operation differentiable and geometrically plausible when surfaces occlude one another. The second component is the adversarial pose generator $G$, a network branch that takes the intermediate feature map and emits a pose vector $p$, clamped by a scaled sigmoid; it is trained with the inverted gradient of $L_{warp}$ plus a quadratic penalty $\lambda(p⊙p)$, so it learns to choose hard views instead of easy ones. During training the depth network receives gradients from both $L_{dep}$ and $L_{warp}$, while the pose generator receives only the adversarial gradient; at inference both the pose generator and the warping branch are discarded.
What would settle it
On NYU Depth v2, take a trained AVCL model, warp its predicted depth and the ground-truth depth with a fixed pose, and for every pixel that contributes to $L_{warp}$ check whether the 3D point that produced the predicted value is the same as the 3D point from the ground-truth depth. If a large share of the high-loss pixels are occlusion or correspondence mismatches, where the predicted depth lands on a different surface than the ground-truth depth at that pixel, then the loss is not enforcing view-consistent geometry as claimed.
Extended reading notes
Core claim
The central claim is that forcing consistency across multiple views is a stronger training signal than the single-view regression loss, and that the views should be chosen adversarially rather than fixed or random. The paper demonstrates this by augmenting existing monocular depth networks with an AVCL branch: the predicted depth map is warped to a target view using a differentiable projection and z-buffering operation $W$, the ground-truth depth map is warped with the same pose, and an $L_1$ loss $L_{warp}$ compares the two warped depth maps. A pose generator $G$ is trained against the depth network by inverting its gradient, so it learns to output a 6-DoF pose that makes the warped prediction disagree most with the warped ground truth. On NYU Depth v2, this raises DORN's $\delta_1$ accuracy from 0.828 to 0.836 with the full training set and improves every tested loss function on the smaller 1.2K training set.
Load-bearing premise
The argument assumes that when both the predicted depth and the ground-truth depth are warped with the same pose, the pixel at a given location in the warped prediction and the pixel at the same location in the warped ground truth correspond to the same physical surface point, so the $L_{warp}$ loss measures a real geometric disagreement rather than comparing unrelated 3D points.
Editorial extensions
If this is right
- Any existing monocular depth network can be augmented with AVCL during training while keeping its inference-time architecture unchanged, so the gain is independent of the network's backbone or loss function.
- Training on one tenth of NYU Depth v2 with AVCL reaches roughly the same accuracy as training on the full dataset without it, indicating that the view-consistency loss acts as data-efficient augmentation.
- The adversarial pose selection matters: fixed poses can hurt performance depending on the pose, and random poses help less than adversarially chosen poses, so the framework transfers the burden of choosing supervision views to the learned pose generator.
- Because the warped branch supervises the depth map as a 3D surface rather than as a pixel array, the predicted depth becomes more consistent across rendering directions, which is exactly what downstream applications such as view synthesis and 3D reconstruction need.
Reading between the lines
- The same adversarial warping idea could apply to other dense prediction tasks with geometric meaning, such as surface-normal estimation or semantic 3D voxel grids, where consistency across views is a natural correctness criterion.
- The paper does not explicitly verify that the warped predicted depth and the warped ground truth refer to the same physical surface point at each pixel; a pixel-level correspondence test would show whether the $L_{warp}$ signal is purely geometric or partly a texture-like averaging effect.
- A testable extension is to replace the single adversarial pose with multiple poses or a sequence of warps and to enforce cycle consistency by warping back and forth, which would reveal whether hard-view mining or sheer view diversity drives the gain.
- Because DORN's ordinal output is approximated with sigmoid sums to make it warpable, the same soft-thresholding trick could extend AVCL to other discrete or ordinal regression networks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an Adversarial View-Consistent Learning (AVCL) framework for monocular depth estimation. The core idea is to warp the predicted depth map into novel views using a proposed differentiable warping operation, compare the warped prediction against a warped ground-truth depth map with an L1 loss, and train an adversarial pose generator to select hard views that maximize this comparison loss. The framework is designed as a plug-in module that can be combined with existing depth estimation losses (L1, berHu, DORN) and networks (ResNet-based). Experiments on NYU Depth v2 are conducted under two training-data settings (full 120K images and a 1/10 down-sampled subset) and show consistent improvements: for example, DORN+AVCL reaches 0.836 δ1 accuracy versus 0.828 for DORN alone when trained on the full dataset, and the method is claimed to reduce the amount of training data needed to reach a given accuracy.
Significance. If the reported results are reliable, the paper makes a useful empirical contribution: a simple, network-agnostic regularization that improves multiple monocular depth baselines, with a tangible data-efficiency gain on a down-sampled training set. The differentiable warping operation and the adversarial hard-view mining idea are interesting building blocks for future view-consistency work. The paper's strengths are the consistent gains across three loss functions and the clear qualitative illustration of view-consistency. However, the paper does not release code or trained models, does not report error bars or multiple runs, and contains internal numerical inconsistencies that currently undermine the credibility of the ablation study.
major comments (3)
- [§4.2, Tables 1 and 4] The ablation table is internally inconsistent with the main comparison table. Table 4 is described as using L1 loss and the down10 training set, but its 'Baseline' entry reports δ1 = 0.817, whereas Table 1 lists the same setting (L1 Loss, no AVCL, down10) as 0.800. Moreover, the 'Adversarial pose' row of Table 4 reports 0.836, which is the same nominal configuration as 'L1 Loss ✓' in Table 1 (0.810). These numbers should coincide unless the two tables use different networks, data splits, or training protocols; the paper does not state any such difference. This discrepancy makes the ablation's conclusions about fixed/random/adversarial poses difficult to interpret and needs to be resolved before the paper can be accepted.
- [§3.1, Eqs. (2)-(3)] The paper's claim that Eq. (3) enforces 'view-consistency' needs a more careful geometric discussion. For a fixed target pixel (i,j), both D'_pred and D'_gt are computed as the minimum depth along the same target-view ray, but that ray hits different scene surfaces depending on the predicted depth. The L1 loss therefore is a valid novel-view depth rendering loss, not a comparison of unrelated scene points; the skeptic's correspondence-mismatch concern does not invalidate the training signal. However, the min-selection in Eq. (2) means that only the point with the smallest z contributes to each target pixel, so depth errors on surfaces that are occluded in the rendered view are never penalized. The paper does not discuss this occlusion/error-hiding behavior or report the fraction of target pixels that are ignored or correspond to different source points. This limitation is directly relevant to the adversarial pose generator, which may exploit occlusion mismatches to inflate L_warp rather than mine geometrically 'hard' views. The authors should add an analysis or at least a discussion of this behavior.
- [§4.2, Table 4] The superiority of the adversarial pose module over random poses rests on a single point estimate: 0.836 versus 0.825 δ1. Given that the fixed-pose ablations show substantial sensitivity (0.802–0.822), a single run is insufficient to support the claim that adversarial mining is necessary or consistently better. The paper should report multiple independent runs with means and standard deviations, or provide a paired comparison over several seeds, for the key ablation rows. Without this, the gain from adversarial pose generation is not statistically grounded.
minor comments (5)
- [Throughout] There are numerous typos and inconsistent notations: 'Tabel' for 'Table', 'A VCL / A VCL' spacing, 'fuctions' for 'functions', 'structral' for 'structural', 'avaible' for 'available', 'contract' for 'contrast', and inconsistent use of 'L1 Loss' versus 'L1 loss'.
- [§4.2, Table 4 caption] The caption 'Comparisons with existing monocular depth estimation methods on NYU Depth v2 dataset' is inaccurate; the table shows ablations of the proposed method and should be relabeled accordingly.
- [§3.1, Eq. (2)] The paper claims the warping operation is differentiable because it uses 'simple value assignment', but the floor operation is non-differentiable at integer boundaries and the min-selection is piecewise constant. The authors should state precisely how gradients are computed through the coordinate discretization, for example by defining an indicator-based straight-through estimator or by noting that gradients flow only to the selected source pixel.
- [§4.1, Implementation Details] The hyperparameter λ is set to (1,1,1,1,1,1) in all experiments, but the paper does not explain whether this vector multiplies the pose vector elementwise or its squared norm; Eq. (4) writes λ·(p⊙p), which is ambiguous if λ is a vector. Clarify the exact regularization form.
- [§4.2, DORN loss adaptation] For the DORN loss, the warped branch supervises a continuous approximation of the thresholded output, while the main branch uses the ordinal regression loss on the thresholded output. The authors should discuss whether the benefit could partly come from a form of auxiliary smoothed-target regression rather than from view-consistency alone.
Circularity Check
No significant circularity: AVCL is an empirical training objective evaluated on an external benchmark; the adversarial pose generator is a min-max mechanism, not a fitted input renamed as a prediction.
full rationale
The paper's chain is empirical rather than derivational: Ldep and Lwarp in Eq. (3) are training losses, and the method's success is measured on NYU Depth v2 with the standard delta, rel, log10, rms, and rms(log) metrics. No parameter is fitted to the evaluation set, and no prediction is defined as the value of a fitted constant. The adversarial pose generator is trained to maximize Lwarp while the depth generator minimizes it; this min-max loop is the proposed mechanism, not a hidden reuse of the output as input. The closest self-referential element is that 'hard views' are those with the largest current warping loss, but that is a training-signal property, not a derivation of the central claim from itself. The concern that D'_pred and D'_gt in Eq. (3) may select different scene points when predicted and ground-truth depths disagree is a correctness and robustness issue about the geometric validity of the supervision, not a circularity: even if the loss is noisy, the benchmark comparison remains external and falsifiable. No load-bearing self-citations or imported uniqueness theorems appear. Thus no circular step meeting the evidence bar is present.
Assumptions & free parameters
free parameters (3)
- λ pose regularization =
(1,1,1,1,1,1)
- Pose sigmoid scale and shift =
unspecified
- DORN soft-threshold width =
100 and 0.5
assumptions (4)
- domain assumption The pinhole camera model with known intrinsics K (Eq. 1) correctly describes image formation for NYU Depth v2.
- domain assumption Ground truth depth maps are accurate enough to serve as warping targets, and invalid/no-depth pixels can be safely ignored.
- ad hoc to paper The min-depth, floor-based warping in Eq. (2) approximates true depth re-rendering and provides useful gradients despite the non-differentiable floor and argmin selection.
- ad hoc to paper The adversarial pose generator, trained with inverted gradients, will converge to hard but non-degenerate views rather than poses that escape the loss by producing mostly ignored pixels.
Cite this review
Pith. "Pith review of Adversarial View-Consistent Learning for Monocular Depth Estimation." pith.science (2026). https://pith.science/paper/RB3PG4Z4
@misc{pith2026190801301,
author = {Pith},
title = {Pith review of: Adversarial View-Consistent Learning for Monocular Depth Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/RB3PG4Z4}},
note = {Machine review of arXiv:1908.01301}
}
read the original abstract
This paper addresses the problem of Monocular Depth Estimation (MDE). Existing approaches on MDE usually model it as a pixel-level regression problem, ignoring the underlying geometry property. We empirically find this may result in sub-optimal solution: while the predicted depth map presents small loss value in one specific view, it may exhibit large loss if viewed in different directions. In this paper, inspired by multi-view stereo (MVS), we propose an Adversarial View-Consistent Learning (AVCL) framework to force the estimated depth map to be all reasonable viewed from multiple views. To this end, we first design a differentiable depth map warping operation, which is end-to-end trainable, and then propose a pose generator to generate novel views for a given image in an adversarial manner. Collaborating with the differentiable depth map warping operation, the pose generator encourages the depth estimation network to learn from hard views, hence produce view-consistent depth maps . We evaluate our method on NYU Depth V2 dataset and the experimental results show promising performance gain upon state-of-the-art MDE approaches.
Figures
Reference graph
Works this paper leans on
-
[1]
Large scale gan training for high fidelity natural image synthesis
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018
arXiv 2018
-
[2]
Depth from a single image by harmonizing overcomplete local network predictions
Ayan Chakrabarti, Jingyu Shao, and Greg Shakhnarovich. Depth from a single image by harmonizing overcomplete local network predictions. In NIPS. 2016
work page 2016
-
[3]
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In CVPR, 2015
work page 2015
-
[4]
Depth map prediction from a single image using a multi-scale deep network
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors,NIPS. 2014
work page 2014
-
[5]
Deep ordinal regression network for monocular depth estimation
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, 2018
2018
-
[6]
Yasutaka Furukawa, Carlos Hernández, et al. Multi-view stereo: A tutorial. F ounda- tions and Trends R© in Computer Graphics and Vision, 9(1-2):1–148, 2015
work page 2015
-
[7]
Massively parallel multi- view stereopsis by surface normal diffusion
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multi- view stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pages 873–881, 2015
2015
-
[8]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014
2014
Show all 42 references
-
[9]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016
2016
-
[10]
Recovering surface layout from an image
Derek Hoiem, Alexei A Efros, and Martial Hebert. Recovering surface layout from an image. International Journal of Computer Vision, 75(1):151–172, 2007
2007
-
[11]
Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis
Rui Huang, Shu Zhang, Tianyu Li, and Ran He. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In ICCV, 2017
2017
-
[12]
Spatial transformer net- works
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer net- works. In NIPS, 2015
2015
-
[13]
Surfacenet: An end-to-end 3d neural network for multiview stereopsis
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In Proceedings of the IEEE International Conference on Computer Vision, pages 2307–2315, 2017
2017
-
[14]
Karsch, C
K. Karsch, C. Liu, and S. B. Kang. Depth transfer: Depth extraction from video us- ing non-parametric sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11):2144–2158, 2014
2014
-
[15]
What uncertainties do we need in bayesian deep learning for computer vision? In nips
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In nips. 2017. Y .LIU, Y .W ANG, S.W ANG: A VCL 11
2017
-
[16]
A theory of shape by space carving
Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. Inter- national journal of computer vision , 38(3):199–218, 2000
2000
-
[17]
Pulling things out of perspective
Lubor Ladicky, Jianbo Shi, and Marc Pollefeys. Pulling things out of perspective. In CVPR, 2014
2014
-
[18]
Deeper depth prediction with fully convolutional residual networks
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 F ourth international conference on 3D vision (3DV), 2016
2016
-
[19]
Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs
Bo Li, Chunhua Shen, Yuchao Dai, Anton van den Hengel, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In CVPR, 2015
2015
-
[21]
A two-streamed network for estimating fine- scaled depth maps from single rgb images
Jun Li, Reinhard Klein, and Angela Yao. A two-streamed network for estimating fine- scaled depth maps from single rgb images. In ICCV, 2017
2017
-
[22]
B. Liu, S. Gould, and D. Koller. Single image depth estimation from predicted semantic labels. In cvpr, 2010
2010
-
[23]
Ce Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William T. Freeman. Sift flow: Dense correspondence across different scenes. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, ECCV, 2008
2008
-
[24]
F. Liu, C. Shen, G. Lin, and I. Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10):2024–2039, 2016
2024
-
[26]
Learning depth from single monocular images using deep convolutional neural fields.IEEE transactions on pattern analysis and machine intelligence, 38(10):2024–2039, 2016
Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning depth from single monocular images using deep convolutional neural fields.IEEE transactions on pattern analysis and machine intelligence, 38(10):2024–2039, 2016
2024
-
[27]
Discrete-continuous depth esti- mation from a single image
Miaomiao Liu, Mathieu Salzmann, and Xuming He. Discrete-continuous depth esti- mation from a single image. In CVPR, 2014
2014
-
[28]
Deep metric learn- ing with bier: Boosting independent embeddings robustly
Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. Deep metric learn- ing with bier: Boosting independent embeddings robustly. IEEE transactions on pat- tern analysis and machine intelligence , 2018
2018
-
[29]
Unsupervised representation learning with deep convolutional generative adversarial networks
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015
2015 arXiv
-
[30]
Monocular depth estimation using neural regres- sion forest
Anirban Roy and Sinisa Todorovic. Monocular depth estimation using neural regres- sion forest. In CVPR, 2016. 12 Y .LIU, Y .W ANG, S.W ANG: A VCL
2016
-
[31]
Saxena, M
A. Saxena, M. Sun, and A. Y . Ng. Make3d: Learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence , 31(5): 824–840, 2009
2009
-
[32]
Learning depth from single monocular images
Ashutosh Saxena, Sung H Chung, and Andrew Y Ng. Learning depth from single monocular images. In NIPS, pages 1161–1168, 2006
2006
-
[33]
Photorealistic scene reconstruction by voxel col- oring
Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel col- oring. International Journal of Computer Vision, 35(2):151–173, 1999
1999
-
[34]
Indoor segmenta- tion and support inference from rgbd images
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmenta- tion and support inference from rgbd images. In ECCV, 2012
2012
-
[35]
Efficient large-scale multi-view stereo for ultra high-resolution image sets
Engin Tola, Christoph Strecha, and Pascal Fua. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Machine Vision and Applications, 23(5):903–920, 2012
2012
-
[36]
Disentangled representation learning gan for pose-invariant face recognition
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In CVPR, 2017
2017
-
[37]
Peng Wang, Xiaohui Shen, Zhe Lin, Scott Cohen, Brian Price, and Alan L. Yuille. Towards unified depth and semantic prediction from a single image. In CVPR, 2015
2015
-
[38]
A-fast-rcnn: Hard positive generation via adversary for object detection
Xiaolong Wang, Abhinav Shrivastava, and Abhinav Gupta. A-fast-rcnn: Hard positive generation via adversary for object detection. In CVPR, 2017
2017
-
[39]
Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks
Junyuan Xie, Ross Girshick, and Ali Farhadi. Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks. In ECCV, 2016
2016
-
[40]
Multi-scale continuous crfs as sequential deep networks for monocular depth estimation
Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Multi-scale continuous crfs as sequential deep networks for monocular depth estimation. InCVPR, 2017
2017
-
[41]
Mvsnet: Depth inference for unstructured multi-view stereo
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pages 767–783, 2018
2018
-
[42]
Unsupervised learning of depth and ego-motion from video
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017
2017
-
[43]
Generative visual manipulation on the natural image manifold
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In ECCV, 2016
2016
-
[44]
Indoor scene structure analysis for single image depth estimation
Wei Zhuo, Mathieu Salzmann, Xuming He, and Miaomiao Liu. Indoor scene structure analysis for single image depth estimation. In CVPR, 2015
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.