REVIEW 4 major objections 6 minor 34 references
Leveraging Stable Diffusion for Monocular Depth Estimation via Image Semantic Encoding
T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Replacing CLIP text with SeeCoder image embeddings lifts KITTI monocular depth accuracy to δ1=0.974 and generalizes to rainy and nighttime driving scenes.
desk verdict The core idea is plausible but the main KITTI comparison is invalidated by a non-standard split-image test protocol, so the paper's central claim is currently unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the spatially enhanced SeeCoder. Its frozen SWIN-L backbone and decoder produce multi-scale feature maps; a Query Transformer turns these into 148 semantic vectors of dimension 768, of which 144 are local queries performing cross-attention with the feature maps and 4 are global queries that capture whole-scene context. On top of each SeeCoder transformer layer the paper adds dilated convolutions and a spatial-attention module (channel-pooled features concatenated and convolved into an attention map), and only these added modules are updated during training. The 148 vectors condition a from-scratch denoising UNet through cross-attention at every resolution, while a frozen VAE encoder supplies the latent representation; the UNet is trained to predict the added noise in the latent diffusion objective.
What would settle it
Evaluate the trained model on the KITTI Eigen validation set using whole 1242×375 images, without the left/right split or horizontal-flip fusion, but with the same Garg crop and valid-depth mask, and compare δ1, RMSE, and AbsRel with the numbers in Table III. If the metrics fall to the SeeCoder-only row (δ1≈0.973, RMSE≈2.216) or below the listed AdaBins and ZoeDepth numbers, then the split-fusion protocol rather than the image-semantic encoder is carrying the reported improvement.
Extended reading notes
Core claim
The paper's central claim is that text-free semantic conditioning fixes a real failure mode of diffusion-based depth estimation in complex outdoor scenes. Its Table III reports δ1=0.974, RMSE=2.179, and AbsRel=0.052 on the KITTI Eigen validation split, ahead of the listed AdaBins, DPT, and ZoeDepth numbers. Table I shows that this comes from combining SeeCoder with both spatial modules, since SeeCoder alone reaches δ1=0.973 and either module alone collapses δ1 to 0.876–0.883. On Waymo, the model keeps global RMSE and AbsRel within roughly 10% across normal, rainy, and nighttime scenes, though δ1 falls sharply in rain and night. The paper acknowledges in its conclusion that it does not yet surpass the current state-of-the-art diffusion depth model and that textureless regions remain failure cases.
Load-bearing premise
The reported KITTI numbers rely on a test-time protocol that splits each 1242×375 image into left and right halves, predicts each half with horizontal flip, and fuses the depth maps; the paper's comparison assumes this protocol does not inflate its metrics relative to baselines scored on whole images.
Editorial extensions
If this is right
- Hand-written text prompts are no longer a prerequisite for semantic conditioning in Stable-Diffusion-based depth estimation.
- Spatial enhancement modules for SeeCoder should be used as a pair: using either dilated convolution or spatial attention alone degrades accuracy in the paper's ablation.
- A model trained only on KITTI can be applied directly to Waymo normal, rainy, and nighttime scenes with global error metrics varying by only about 10%.
- Textureless regions such as dark night areas and rain droplets on the lens remain systematic failure cases that future work must address.
- The approach is positioned as a drop-in semantic condition, so other perception heads built on the same UNet-decoder design could adopt image-based semantics.
Reading between the lines
- The split-and-fuse evaluation protocol is the most direct alternative explanation for the KITTI margin; comparing whole-image scores would separate the encoder's contribution from the test-time augmentation's contribution.
- Because SeeCoder emits 144 local queries, the model is effectively learning a per-image set of visual prompts; varying that number would reveal how much semantic capacity depth estimation actually needs.
- The same image-semantic conditioning should transfer to other dense prediction tasks currently tied to CLIP text prompts, such as referring segmentation or open-vocabulary detection, where scene captions are equally hard to generate.
- The paired-module ablation suggests a caution for future work: adding a single spatial enhancement to a frozen semantic encoder can be destabilizing, so module interactions should be tested jointly rather than independently.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes replacing CLIP text embeddings with SeeCoder image-based semantic embeddings in the VPD Stable Diffusion framework for monocular depth estimation, adding a spatial enhancement module composed of dilated convolutions and spatial attention. The method is evaluated on the KITTI Eigen split and on three Waymo Open Dataset scenarios (normal, rainy, nighttime). The authors report KITTI validation metrics of δ1=0.974, RMSE=2.179, AbsRel=0.052, claim performance comparable to state-of-the-art models, and interpret the Waymo results as evidence of robustness across weather and lighting conditions.
Significance. The underlying idea—using image-derived semantic embeddings instead of text prompts in a diffusion-based depth estimator—is motivated by a real limitation of CLIP text conditioning in complex outdoor scenes. If the reported gains were reliable, the paper would provide a simple and potentially transferable alternative for dense prediction tasks. However, the empirical support is severely undercut by the non-standard KITTI evaluation protocol, the absence of the direct VPD/CLIP baseline, and Waymo numbers that contradict the robustness claim. The contribution is incremental: SeeCoder and the VPD framework are imported from prior work, and the only novel component is the spatial enhancement module, whose ablation shows individual modules hurt performance and whose combined gain is marginal. Consequently, the paper's significance as a standalone contribution is limited.
major comments (4)
- [Section IV-C and Table III] The KITTI comparison is compromised by a non-standard test-time protocol. The paper states that during evaluation the 1242×375 images are split into left and right segments, each segment is predicted with horizontal-flip augmentation, and the two segments are "weighted and fused." All compared baselines in Table III (Eigen, MonoDepth2, AdaBins, DPT, ZoeDepth) are scored on whole images using the standard Eigen evaluation. The split changes the input aspect ratio and receptive field, and the flip-and-fusion procedure is a form of test-time ensembling that is not granted to the baselines. No whole-image result for the proposed model is reported, and the fusion weights are not specified. As the only quantitative support for the central claim of state-of-the-art-comparable performance is this protocol-dependent KITTI result, the comparative claim is not verifiable as presented.
- [Section IV-E and Table III] The paper's central motivation is that SeeCoder image embeddings outperform CLIP text embeddings in outdoor scenes, but Table III does not include VPD with CLIP on KITTI. Since the proposed method is exactly the VPD framework with the semantic encoder swapped and a spatial module added, the missing VPD/CLIP baseline is load-bearing: without it, the reader cannot attribute any observed gain to the image-based semantic encoding rather than to other differences in training or evaluation. The paper should report the VPD model (or an equivalent CLIP-conditioned Stable Diffusion depth model) under the same protocol and training setup.
- [Section IV-D and Table II] The Waymo results contradict the robustness claim. Table II reports δ1 values of 0.044 (normal), 0.012 (rainy), and 0.017 (nighttime); in daytime conditions similar to KITTI, 95.6% of pixels are outside the 1.25× accuracy threshold. The text interprets the relatively small variation in RMSE and AbsRel across scenes as evidence of "generalization and robustness," but the absolute error levels are so high that the model is effectively failing on this dataset. At minimum, the paper should acknowledge that the model does not transfer to Waymo and should not claim robustness on the basis of these numbers.
- [Section III-B and Table I] The ablation study does not support the claim that the spatial enhancement module improves performance. Adding dilated convolution (DC) alone drops δ1 from 0.973 to 0.876 and increases Sq Rel from 0.164 to 1.155; adding spatial attention (SA) alone drops δ1 to 0.883. The combined DC+SA yields δ1=0.974 versus 0.973 and RMSE=2.179 versus 2.216, a marginal difference with no significance testing. The proposed explanation that "their combination compensates for each other's shortcomings" is speculative. Given that each individual module is clearly harmful, the paper should provide a more thorough analysis and preferably report standard deviations or multiple seeds.
minor comments (6)
- [References [32]] The paper attributes adaptive binning to Tu et al. [32], but the cited reference is about binary neural networks (AdaBin), not the AdaBins depth estimation method. Please cite the original AdaBins paper (Bhat et al., CVPR 2021).
- [Abstract and Conclusion] The abstract claims performance "comparable to state-of-the-art models," while the conclusion states the method "does not yet surpass current state-of-the-art models" and cites ECoDepth. Please reconcile these statements and specify which state-of-the-art models are being compared.
- [Section IV-C] The weighted fusion of the left and right segments is described only as "weighted and fused." Please specify the weights or the rule used to compute them, as this is necessary for reproducibility.
- [Section III-C] The statement that the denoising UNet is "trained from scratch without relying on pre-trained weights" is ambiguous because the VAE and SeeCoder backbones are pre-trained. Clarify whether only the UNet parameters are randomly initialized.
- [Equation (2)] The notation \bar{α}_t is used without a definition of the noise schedule; please define α_t and \bar{α}_t.
- [Section II-C] There is a typo: "Semantic encoders play a important role" should read "an important role."
Circularity Check
No significant circularity: the semantic encoder and diffusion backbone are external, the claimed gains are measured against ablations and published baselines, and the paper states its own limitations.
full rationale
Walked the claimed derivation chain end to end. The semantic encoder is SeeCoder, imported from the external Prompt-free Diffusion paper [14]; the Stable Diffusion/VPD framework is also external [12]; the spatial enhancement modules are standard dilated convolutions and spatial attention whose effect is measured in Table I against a SeeCoder-only baseline; and the final headline numbers in Table III are empirical validation-set measurements compared with published numbers from Eigen, Monodepth2, AdaBins, DPT, and ZoeDepth. No step defines the output in terms of the input, no fitted parameter is renamed as a prediction, and no load-bearing uniqueness claim is imported from the authors' own prior work. The most serious concern is in Section IV-C: the paper evaluates on KITTI by splitting each 1242x375 image into left and right segments, applying horizontal-flip augmentation to each segment, and fusing the two predictions, while the cited baselines are scored on whole images. That is a genuine experimental-validity and comparability issue, but it is not circularity: the reported numbers are still measurements of the model, not consequences of its definitional setup. The paper also explicitly limits its own claim in Section V: 'our model struggles with predicting depth in textureless regions and does not yet surpass current state-of-the-art models [9]'. This honest limitation further confirms that the central claim is an empirical result with independent content, not a tautology or a self-citation chain. No circular step can be exhibited with a quoted equation or fitted-parameter reduction, so the appropriate finding is no significant circularity with score 0.
Assumptions & free parameters
free parameters (1)
- Left/right segment fusion weights =
not specified
assumptions (4)
- domain assumption Stable Diffusion latent diffusion provides a suitable backbone for dense prediction.
- ad hoc to paper SeeCoder's frozen image-based embeddings carry semantic information at least as useful for depth estimation as CLIP's text-based embeddings.
- domain assumption The Garg crop and Eigen split are appropriate evaluation protocols for KITTI.
- ad hoc to paper The left/right image splitting and weighted fusion does not bias the comparison against baselines.
Cite this review
Pith. "Pith review of Leveraging Stable Diffusion for Monocular Depth Estimation via Image Semantic Encoding." pith.science (2026). https://pith.science/paper/FEXM6KFK
@misc{pith2026250201666,
author = {Pith},
title = {Pith review of: Leveraging Stable Diffusion for Monocular Depth Estimation via Image Semantic Encoding},
year = {2026},
howpublished = {\url{https://pith.science/paper/FEXM6KFK}},
note = {Machine review of arXiv:2502.01666}
}
read the original abstract
Monocular depth estimation involves predicting depth from a single RGB image and plays a crucial role in applications such as autonomous driving, robotic navigation, 3D reconstruction, etc. Recent advancements in learning-based methods have significantly improved depth estimation performance. Generative models, particularly Stable Diffusion, have shown remarkable potential in recovering fine details and reconstructing missing regions through large-scale training on diverse datasets. However, models like CLIP, which rely on textual embeddings, face limitations in complex outdoor environments where rich context information is needed. These limitations reduce their effectiveness in such challenging scenarios. Here, we propose a novel image-based semantic embedding that extracts contextual information directly from visual features, significantly improving depth prediction in complex environments. Evaluated on the KITTI and Waymo datasets, our method achieves performance comparable to state-of-the-art models while addressing the shortcomings of CLIP embeddings in handling outdoor scenes. By leveraging visual semantics directly, our method demonstrates enhanced robustness and adaptability in depth estimation tasks, showcasing its potential for application to other visual perception tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Enhanced Computer Vision With Microsoft Kinect Sensor: A Review,
J. Han, L. Shao, D. Xu, and J. Shotton, “Enhanced Computer Vision With Microsoft Kinect Sensor: A Review,” IEEE transactions on cybernetics , vol. 43, no. 5, pp. 1318-1334, Oct. 2013
work page 2013
-
[2]
Deep Learning for Monocular Depth Estimation: A Review,
Y . Ming, X. Meng, C. Fan, and H. Yu, “Deep Learning for Monocular Depth Estimation: A Review,” Neurocomputing, vol. 438, pp. 14-33, May 2021
work page 2021
-
[3]
Monocular depth prediction using generative adversarial networks
C.S. Kumar, S.M. Bhandarkar, and M. Prasad. "Monocular depth prediction using generative adversarial networks." In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 300-308. 2018
work page 2018
-
[4]
Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network,
D. Eigen, C. Puhrsch, and R. Fergus, “Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network,” in Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), vol. 2, pp. 2366–2374, Dec. 2014
work page 2014
-
[5]
Depth Prediction from a Single Image with Conditional Adversarial Networks,
H. Jung, Y . Kim, C. Oh, and K. Sohn, “Depth Prediction from a Single Image with Conditional Adversarial Networks,” in Proceedings of the IEEE International Conference on Image Processing (ICIP) , Beijing, 2017, pp. 1717-1721
work page 2017
-
[6]
Improved Techniques for Training GANs,
T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, “Improved Techniques for Training GANs,” in Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), pp. 2234–2242, Dec. 2016
work page 2016
-
[7]
Diffusion Models Beat GANs on Image Synthesis,
P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” in Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS) , vol. 1, Article No. 672, pp. 8780–8794, Jun. 2024
work page 2024
-
[8]
Monocular Depth Es- timation using Diffusion Models,
S. Saxena, A. Kar, M. Norouzi, and D. J. Fleet, “Monocular Depth Es- timation using Diffusion Models,” arXiv preprint, arXiv:2302.14816, Feb. 2023
arXiv 2023
Show all 34 references
-
[9]
ECoDepth: Effective Condi- tioning of Diffusion Models for Monocular Depth Estimation,
S. Patni, A. Agarwal, and C. Arora, “ECoDepth: Effective Condi- tioning of Diffusion Models for Monocular Depth Estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28285–28295, Jun. 2024
2024
-
[10]
Denoising Diffusion Probabilistic Models,
J. Ho, A. Jain, and P. Abbeel, “Denoising Diffusion Probabilistic Models,” in Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS) , vol. 1, Article No. 574, pp. 6840–6851, Dec. 2020
2020
-
[11]
High-Resolution Image Synthesis with Latent Diffusion Models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10694, 2022
2022
-
[12]
Unleashing text-to-image diffusion models for visual perception
W. Zhao, R. Yongming, Z. Liu, B. Liu, J. Zhou, and J. Lu. "Unleashing text-to-image diffusion models for visual perception." In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 5729-5739. 2023
2023
-
[13]
Learning transferable visual models from natural language supervision
A. Radford, J.W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, and G. Krueger, "Learning transferable visual models from natural language supervision." In International conference on machine learning, pp. 8748-8763. PMLR, 2021
2021
-
[14]
Prompt- free diffusion: Taking
X. Xu, J. Guo, Z. Wang, G. Huang, I. Essa, and H. Shi, "Prompt- free diffusion: Taking" text" out of text-to-image diffusion models." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8682-8692. 2024
2024
-
[15]
Vision Meets Robotics: The KITTI Dataset,
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision Meets Robotics: The KITTI Dataset,” Int. J. Robot. Res., vol. 32, no. 11, pp. 1231-1237, Sep. 2013
2013
-
[16]
Scalability in perception for autonomous driving: Waymo open dataset
P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V . Patnaik, P. Tsui, J. Guo, Y . Zhou, Y . Chai, B. Caine and V . Vasudevan. "Scalability in perception for autonomous driving: Waymo open dataset." In Proceedings of the IEEE/CVF conference on computer vision and pattern reco...
2020
-
[17]
Digging into Self-Supervised Monocular Depth Prediction,
C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into Self-Supervised Monocular Depth Prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 2019
2019
-
[18]
Generative Adversarial Nets,
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, "Generative Adversarial Nets," in Advances in Neural Information Processing Systems 27 (NIPS) , 2014, pp. 2672–2680
2014
-
[19]
Wasserstein GAN,
M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” arXiv preprint, arXiv:1701.07875, Dec. 2017
2017 arXiv
-
[20]
Large Scale GAN Training for High Fidelity Natural Image Synthesis,
A. Brock, J. Donahue, and K. Simonyan, “Large Scale GAN Training for High Fidelity Natural Image Synthesis,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2019
2019
-
[21]
Alias-Free Generative Adversarial Networks,
T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila, “Alias-Free Generative Adversarial Networks,” in Pro- ceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS) , 2021
2021
-
[22]
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,
J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” in Proceedings of the 40th International Confer- ence on Machine Learning (ICML) , vol. 2023, Article No. 814, pp. 19730–19742, Jul. 2023
2023
-
[23]
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,
A. Dosovitskiy, et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint , arXiv:2010.11929, 2021
2010 arXiv
-
[24]
Laion-400m: Open Dataset of CLIP-filtered 400 Million Image-Text Pairs,
C. Schuhmann et al., “Laion-400m: Open Dataset of CLIP-filtered 400 Million Image-Text Pairs,” arXiv preprint, arXiv:2111.02114, 2021
2021 arXiv
-
[25]
Coyo-700m: Image-text Pair Dataset,
M. Byeon, B. Park, H. Kim, S. Lee, W. Baek, and S. Kim, “Coyo-700m: Image-text Pair Dataset,” 2022. https://github.com/kakaobrain/coyo-dataset. Accessed: Aug. 26, 2024
2022
-
[26]
CBAM: Convolutional Block Attention Module,
S. Woo, J. Park, J.-Y . Lee, and I. S. Kweon, “CBAM: Convolutional Block Attention Module,” in Proceedings of the 15th European Con- ference on Computer Vision (ECCV) , Munich, Germany, Sep. 2018, pp. 3–19
2018
-
[27]
Multi-Scale Context Aggregation by Dilated Convolutions,
F. Yu and V . Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2016
2016
-
[28]
Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows," in In Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 9992–10002
2021
-
[29]
Attention is All You Need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is All You Need," in Advances in Neural Information Processing Systems (NIPS) , 2017, pp. 6000–6010
2017
-
[30]
PyTorch: An Imperative Style, High-Performance Deep Learning Library,
A. Paszke et al. , “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in neural information processing systems(NeurIPS), 2019
2019
-
[31]
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue,
R. Garg, B. G. V . Kumar, G. Carneiro, and I. Reid, "Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue," in Proc. Eur. Conf. Comput. Vis. (ECCV) , 2016, pp. 740–756
2016
-
[32]
AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets,
Z. Tu, X. Chen, P. Ren, and Y . Wang, “AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets,” in Proc. Eur. Conf. Comput. Vis. (ECCV) , Cham, 2022, vol. 13671, pp. 379-395
2022
-
[33]
Vision transformers for dense prediction
R. Ranftl, A. Bochkovskiy and V . Koltun, "Vision transformers for dense prediction." In Proceedings of the IEEE/CVF international conference on computer vision , pp. 12179-12188. 2021
2021
-
[34]
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth,
S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller, “ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth,” arXiv preprint, arXiv:2302.12288, 2023
2023 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.