Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Toward Better SSIM Loss for Unsupervised Monocular Depth Estimation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing multiplicative SSIM with an additive form smooths gradients and improves unsupervised monocular depth estimation on KITTI.

desk verdict Plausible additive-SSIM loss for unsupervised depth, undermined by weight selection on the test split and small, inconsistent gains. read the letter →

arxiv 2506.04758 v1 pith:TNZGH6GK submitted 2025-06-05 cs.CV

Yijun Cao , Fuya Luo , Yongjie Li This is my paper
classification cs.CV
keywords monoculardepthestimationunsupervisedlearningSSIMlossphotometricconsistencysub-pixelconvolutionKITTIdatasetgradientsmoothnessself-supervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that the conventional way of building an SSIM-based photometric loss — multiplying the luminance, contrast, and structure terms together — is a poor training signal for unsupervised monocular depth estimation, and that adding the same three terms with tuned weights instead produces smoother gradients and better depth maps. The proposed additive SSIM is combined with mean absolute error into a single loss, and used to replace the photometric loss inside two existing models, Monodepth2 and GVO, trained on KITTI. With weights $w_1=0.4$, $w_l=0.5$, $w_c=0.5$, $w_s=0.7$ and sub-pixel convolution for upsampling, both models improve on squared relative error and RMSE while keeping absolute relative error roughly unchanged. If this holds, it means part of the gap between supervised and unsupervised depth learning can be closed by reshaping the loss rather than by adding network modules or geometric constraints.

What carries the argument

The load-bearing mechanism is the additive SSIM loss, Eq. (9), and its MAE combination, Eq. (11). Where the standard SSIM multiplies luminance, contrast, and structure, so that the gradient shrinks wherever any factor saturates, the additive form is a weighted sum of distances-to-one for the three components, giving each a contribution to the gradient that is independent of the others' values. This is what the authors mean by smoother gradients: the loss surface no longer has flat plateaus that stall training. The second component is sub-pixel convolution, which rearranges channel dimensions into spatial dimensions to upsample the depth map, replacing nearest interpolation at each decoder scale; the paper reports that this gives consistent performance even at 1/16 resolution.

What would settle it

Select the loss weights on a validation split disjoint from the KITTI test set, then evaluate on that test set: if additive SSIM no longer outperforms the multiplicative baseline by a comparable margin, the published gains are an artifact of choosing weights on the test set. Optionally, measure gradient-norm variance of the two loss forms at fixed random weights to test the smoother-gradient mechanism directly.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that SSIM's three components — luminance $L$, contrast $C$, and structure $S$ — should be combined by addition rather than multiplication when SSIM is used as a training loss. The multiplicative form $1 - L^\alpha C^\beta S^\gamma$ produces gradients whose size depends on the product's curvature, so large flat regions of the loss carry almost no learning signal; the additive form $SSIM_a = w_l(1-L) + w_c(1-C) + w_s(1 - \tfrac{1}{2}(1+S))$ gives each component a gradient contribution that does not vanish when the others are near one. Combined with MAE as $L_a = w_1 MAE + w_l(1-L) + w_c(1-C) + w_s(1 - \tfrac{1}{2}(1+S))$, the loss is optimized with grid-selected weights $w_1=0.4$, $w_l=0.5$, $w_c=0.5$, $w_s=0.7$. Plugging this loss, and sub-pixel convolution in place of nearest-neighbor upsampling, into Monodepth2 improves SqRel from 0.903 to 0.770 and RMS from 4.863 to 4.813 on the KITTI test split; the same recipe improves GVO from SqRel 0.787 to 0.747 and RMS 4.488 to 4.416. The authors explicitly note the gains are modest and that the $\delta_1<1.25$ accuracy slightly drops with sub-pixel upsampling because the smoother depth lacks edge precision.

Load-bearing premise

The reported gains come from weights selected by grid search on the same KITTI test split whose metrics are later reported as the results, and the assumption is that these test-chosen weights improve depth on new scenes and datasets rather than merely fitting that evaluation split.

Editorial extensions

If this is right

  • Substituting Eq. (9) into the photometric loss of an existing unsupervised depth model should improve SqRel and RMS without changing the network; the paper demonstrates this on Monodepth2 and GVO.
  • The weight ranking ($w_c=0.5$, $w_s=0.7$ versus $w_l=0.5$) indicates contrast and structure are the informative parts of SSIM for depth training, so future loss designs should weight them at least as heavily as luminance.
  • Sub-pixel convolution helps most on error metrics sensitive to large depth mistakes and slightly hurts the $\delta_1$ accuracy; applications that need edge precision should weigh this trade-off.
  • The authors claim the additive loss is compatible with occlusion masking and pose refinement, so it can be dropped into other self-supervised SIDE pipelines without violating their assumptions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The loss weights were chosen by grid search on the same KITTI test split whose metrics are reported; a fairer evaluation would select weights on a separate validation split, and the current margins might shrink under that protocol.
  • The smoother-gradient claim is testable directly: at matched random initializations, the additive loss should show lower variance in gradient norms across pixels than the multiplicative loss; the paper does not report such a measurement.
  • Because sub-pixel upsampling blurs edges, a hybrid decoder — sub-pixel in smooth regions, nearest or learned upsampling near depth discontinuities — could recover the lost $\delta_1$ accuracy while keeping the SqRel gains.
  • The same additive-SSIM idea could apply to other self-supervised photometric tasks (optical flow, video prediction), but the required weights may differ and the paper does not test them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a modified SSIM loss for unsupervised monocular depth estimation. Instead of the standard multiplicative combination of luminance, contrast, and structure terms, Eq. (9) combines them additively, and Eq. (11) fuses this additive SSIM with an MAE term using weights w1, wl, wc, ws. The paper also replaces nearest-neighbor upsampling with sub-pixel convolution in the multi-scale decoder. Experiments on the KITTI Eigen split with Monodepth2 and GVO report improvements on SqRel, RMS, and RMSlog, while AbsRel and δ1 are often unchanged or worse. The central claim is that the additive SSIM leads to smoother gradients and higher performance, and that the optimized loss plus sub-pixel convolution 'remarkably outperform[s]' the baseline.

Significance. If validated, the proposed additive SSIM is a simple, inexpensive modification that could transfer to many self-supervised depth pipelines. The paper has several strengths: it conducts an extensive ablation over many weight combinations in Tables 1 and 2, tests two different backbone methods (Monodepth2 and GVO), reports results at multiple decoder resolutions in Table 4, and includes an explicit limitation section. However, the current evidence is not yet convincing because the final reported weights are selected on the same test split used for evaluation, no error bars or repeated runs are provided, and the paper's own numbers show only small and partly mixed improvements. The central empirical claim therefore needs stronger support before the conclusion can be accepted.

major comments (4)
  1. [§4.1, Tables 2 and 5] The evaluation protocol is circular in the load-bearing step: the weights w1=0.4, wl=0.5, wc=0.5, ws=0.7 used in Table 5 are the best configuration found by the grid search in Table 2, and Table 2 reports metrics on the same Eigen test split that is later used for the final comparison. Selecting the best of roughly twenty configurations on the test set introduces optimistic bias that is not accounted for. The paper should either perform weight selection on a held-out validation split and then report test results, or show that the selected weights transfer across multiple seeds and, ideally, to a different dataset or split.
  2. [§4, Implementation details; Tables 2 and 3] All experiments use a single fixed random seed, and no error bars or repeated runs are reported. The differences between many configurations in Table 2 are extremely small (e.g., AbsRel 0.114 vs 0.115, RMS 4.822 vs 4.823, RMSlog 0.191 vs 0.192). Without an estimate of training run-to-run variance, the ranking of configurations and even the claimed improvement over the baseline cannot be distinguished from noise. At minimum, three or more seeds per configuration and mean±std reporting are needed for the principal comparisons.
  3. [§5, Limitation paragraph; Abstract; Table 5] The abstract's claim that the method can 'remarkably outperform' the baseline is not supported by the paper's own numbers. The Limitation section reports improvements of only 0.7% in RMS (4.856 to 4.822) and 6.4% in SqRel (0.868 to 0.816), and Table 5 shows that Monodepth2 + SSC leaves AbsRel unchanged at 0.115 while worsening δ1 (0.863 vs 0.877) and δ2 (0.957 vs 0.961) relative to Monodepth2. The paper should temper the wording to 'improves selected error metrics' and discuss the accuracy metrics that are not improved.
  4. [§3.2 and Figure 1] The central motivation that additive combination yields 'smoother gradients' and avoids convergence problems is not directly tested. Figure 1 illustrates the functional forms, and Eq. (9) gives constant per-component gradients, but the paper provides no gradient statistics, loss curves, or convergence diagnostics during training. The empirical results could still support the proposal, but the gradient-smoothness mechanism remains an unverified hypothesis rather than a demonstrated property of the training process.
minor comments (4)
  1. [Throughout] The manuscript contains several typos and grammatical errors, including 'comparied' (Section 1), 'Comparision' (Section 4), 'the the weights' (Section 4.2), 'futher' (Section 5), and 'Entensive' (Section 6). A careful proofreading pass is needed.
  2. [Eqs. (8) and (9)] The notation for the structure component is confusing: S is stated to lie in [−1,1], but Eq. (9) writes (1 − 1/2(1+S)), which equals (1−S)/2. Defining a normalized structure term explicitly would make the relationship between Eqs. (8) and (9) clearer.
  3. [Tables 1 and 5] The reported numbers for Monodepth2 [7] differ between Table 1 and Table 5 (e.g., δ2 0.959 vs 0.961, δ3 0.981 vs 0.982). The authors should clarify whether these are different runs or different published sources, and use consistent baseline values throughout.
  4. [§4.2] The text refers to 'our model' when describing GVO + SSC, but GVO is the method of [3]; the contribution here is limited to the loss and the upsampling module. Please clarify the scope of the proposed changes for each backbone.

Circularity Check

1 steps flagged · score 6.0 of 10

Final loss weights are tuned on the Eigen test split and then reported as predictions on the same split, making the headline gains a selection artifact.

  1. fitted input called prediction [Section 4, Implementation details; Eq. (11); Tables 2, 4, 5]
    "For depth, training was done on the KITTI raw [1,7] and the frames were resized to 640×192 pixels. The depth was evaluated on the Eigen’s testing split [5]. ... Table 2 shows the experimental results. ... Note that the the weights of SSIMa is set to w1 = 0.4, wl = 0.5, wc = 0.5, ws = 0.7 in Table 5."

    Eq. (11) has tunable weights w1, wl, wc, ws. The paper sweeps these weights in Table 2 and reports each configuration's depth metrics on the Eigen testing split, then picks the configuration (0.4, 0.5, 0.5, 0.7) for the final Tables 3-5. Because the same Eigen test split is used both for model selection and for the reported final metrics, the headline gains are not out-of-sample predictions: they are the best-of-sweep numbers on the test set. The claim that the optimized loss 'remarkably outperform[s] the baseline' is therefore statistically forced by the selection procedure, not demonstrated by an independent evaluation.

full rationale

The additive SSIM in Eq. (9) is a definition, and Eq. (11) is a loss proposal; neither is derived from the target result, so those parts are not circular. The central load-bearing empirical claim, however, is not self-contained: the final weights are chosen by grid search on the same KITTI Eigen split used to report the final depth metrics. That is a fitted-input-called-prediction pattern: the test set functions as a validation set, and the reported improvement is best-of-many test metrics. The paper contains self-citations (e.g., GVO [3]) but they serve as baselines, not as load-bearing justifications for the loss form. No uniqueness theorem or ansatz-via-citation pattern applies. Because the primary quantitative claim reduces to test-set selection, the score is 6.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The loss weights w1, wl, wc, and ws are the paper's free parameters; they are tuned against the test split. The main domain axiom is that photometric loss approximates depth quality; the main ad hoc axiom is that additive SSIM improves gradient flow.

free parameters (4)
  • w1 (MAE weight) = 0.4 (selected in Table 2 grid search)
    Weight on the MAE term in Eq. (11); chosen by grid search on the KITTI Eigen test split and used in the final SSC model.
  • wl (luminance weight) = 0.5
    Weight on the luminance term in Eq. (11); chosen by grid search on the same test split.
  • wc (contrast weight) = 0.5
    Weight on the contrast term in Eq. (11); chosen by grid search on the same test split.
  • ws (structure weight) = 0.7
    Weight on the structure term in Eq. (11); chosen by grid search on the same test split.
assumptions (3)
  • domain assumption Photometric consistency between temporally adjacent frames is a valid training signal for geometric depth.
    Invoked in Section 3.1, Eq. (4); the whole pipeline assumes warping error correlates with depth accuracy.
  • ad hoc to paper Additive combination of SSIM components yields smoother gradients and better optimization than multiplicative combination.
    Asserted in Section 3.2 after Eq. (9) and supported only by toy curves in Figure 1; no proof or rigorous link to convergence is given.
  • domain assumption The KITTI Eigen test split is representative of out-of-sample performance for selecting loss hyperparameters.
    Used in Section 4.1 grid searches and Section 4.2 final results; weights are chosen on this split and then reported as the outcome.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Better SSIM Loss for Unsupervised Monocular Depth Estimation." pith.science (2026). https://pith.science/paper/TNZGH6GK

@misc{pith2026250604758,
  author       = {Pith},
  title        = {Pith review of: Toward Better SSIM Loss for Unsupervised Monocular Depth Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNZGH6GK}},
  note         = {Machine review of arXiv:2506.04758}
}
read the original abstract

Unsupervised monocular depth learning generally relies on the photometric relation among temporally adjacent images. Most of previous works use both mean absolute error (MAE) and structure similarity index measure (SSIM) with conventional form as training loss. However, they ignore the effect of different components in the SSIM function and the corresponding hyperparameters on the training. To address these issues, this work proposes a new form of SSIM. Compared with original SSIM function, the proposed new form uses addition rather than multiplication to combine the luminance, contrast, and structural similarity related components in SSIM. The loss function constructed with this scheme helps result in smoother gradients and achieve higher performance on unsupervised depth estimation. We conduct extensive experiments to determine the relatively optimal combination of parameters for our new SSIM. Based on the popular MonoDepth approach, the optimized SSIM loss function can remarkably outperform the baseline on the KITTI-2015 outdoor dataset.

Figures

Figures reproduced from arXiv: 2506.04758 by the authors.

Figure 1
Figure 1. Four toy examples to illustrate the effect of different components of the SSIM on the output. x indicates luminance and contrast parts, whose outputs are in the range [0, 1]. y indicates structure part, whose output is in the range [−1, 1]. where κ is the weight and usually set to 0.85 [13, 7]. SSIM is the function of structure similarity index measure [24] for evaluating the similarity between two images, which is … view at source ↗
Figure 2
Figure 2. The network architecture. (a) the nearest upsampling module used in the base￾line network.. (b) the sub-pixel convolution upsampling module used in our network. 3.3 Multi-scale Depth Estimation The depth network is similar to the architecture in [7], which adopts encoder￾decoder design with skip connections and five-scale side outputs. The encoder is ResNet18 [9] without full connection layers; at each scale, the de… view at source ↗
Figure 3
Figure 3. Visualization of different upsampling methods [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of different upsampling methods at different resolution. 5 Discussion Motivation. From the first unsupervised SIDE approach [23] to recent state￾of-the-art methods [3, 7], the combination of MAE with SSIM as loss function seems to be a standard option. Ho…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [1]

    In: NeurIPS (2019)

    Bian, J.W., Li, Z., Wang, N., Zhan, H., Shen, C., Cheng, M.M., Reid, I.: Unsu- pervised scale-consistent depth and ego-motion learning from monocular video. In: NeurIPS (2019)

    work page 2019

  2. [2]

    IEEE T-MM23, 761–771 (2021)

    Cao, Y.J., Lin, C., Li, Y.J.: Learning crisp boundaries using deep refine- ment network and adaptive weighting loss. IEEE T-MM23, 761–771 (2021). https://doi.org/10.1109/TMM.2020.2987685

    arXiv 2021

  3. [3]

    Pattern Recognition136, 109262 (2023)

    Cao, Y.J., Zhang, X.S., Luo, F.Y., Peng, P., Lin, C., Yang, K.F., Li, Y.J.: Learning generalizedvisualodometryusingposition-awareopticalflowandgeometricbundle adjustment. Pattern Recognition136, 109262 (2023)

    work page 2023

  4. [4]

    In: ICCV (2015)

    Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: ICCV (2015)

    work page 2015

  5. [5]

    In: NeurIPS (2014)

    Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: NeurIPS (2014)

    2014

  6. [6]

    IJRR (2013)

    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. IJRR (2013)

    work page 2013

  7. [7]

    In: ICCV (2019) 12 Y

    Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self- supervised monocular depth prediction. In: ICCV (2019) 12 Y. Cao et al

    work page 2019

  8. [8]

    Nature585(7825), 357–362 (2020)

    Harris, C.R., Millman, K.J., Van Der Walt, S.J., Gommers, R., Virtanen, P., Cour- napeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., et al.: Array programming with numpy. Nature585(7825), 357–362 (2020)

    work page 2020

Show all 27 references
  1. [9]

    In: CVPR (2016)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

    2016

  2. [10]

    In: WACV

    Hu, J., Ozay, M., Zhang, Y., Okatani, T.: Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In: WACV. pp. 1043–1051. IEEE (2019)

    2019

  3. [11]

    In: ICLR (2015)

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)

    2015

  4. [12]

    In: ICLR (2023)

    Liu, C., Kumar, S., Gu, S., Timofte, R., Van Gool, L.: Va-depthnet: A variational approach to single image depth prediction. In: ICLR (2023)

    2023

  5. [13]

    IEEE T-PAMI42(10), 2624–2641 (2020)

    Luo, C., Yang, Z., Wang, P., Wang, Y., Xu, W., Nevatia, R., Yuille, A.: Every pixel counts ++: Joint learning of geometry and motion with 3d holistic understanding. IEEE T-PAMI42(10), 2624–2641 (2020). https://doi.org/10.1109/TPAMI.2019.2930258

    2020

  6. [14]

    In: NeurIPS (2019)

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. In: NeurIPS (2019)

    2019

  7. [15]

    International Journal of Computer Vision129(8), 2352–2374 (2021)

    Peng, P., Yang, K.F., Luo, F.Y., Li, Y.J.: Saliency detection inspired by topological perception theory. International Journal of Computer Vision129(8), 2352–2374 (2021)

    2021

  8. [16]

    In: CVPR (2019)

    Ranjan, A., Jampani, V., Balles, L., Sun, D., Kim, K., Wulff, J., Black, M.J.: Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In: CVPR (2019)

    2019

  9. [17]

    In: CVPR

    Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR. pp. 1874–1883 (2016)

    2016

  10. [18]

    IEEE Transactions on Circuits and Systems for Video Technology (2022)

    Wang, G., Zhong, J., Zhao, S., Wu, W., Liu, Z., Wang, H.: 3d hierarchical refine- ment and augmentation for unsupervised learning of depth and pose from monocu- lar video. IEEE Transactions on Circuits and Systems for Video Technology (2022)

    2022

  11. [19]

    In: CVPR (2020)

    Yang, N., Stumberg, L., Wang, R., Cremers, D.: D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In: CVPR (2020)

    2020

  12. [20]

    In: CVPR (2018)

    Yin, Z., Shi, J.: Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In: CVPR (2018)

    2018

  13. [21]

    In: CVPR

    Yuan, W., Gu, X., Dai, Z., Zhu, S., Tan, P.: Neural window fully-connected crfs for monocular depth estimation. In: CVPR. pp. 3916–3925 (2022)

    2022

  14. [22]

    In: CVPR (2020)

    Zhao, W., Liu, S., Shu, Y., Liu, Y.J.: Towards better generalization: Joint depth- pose learning without posenet. In: CVPR (2020)

    2020

  15. [23]

    In: CVPR (2017)

    Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: CVPR (2017)

    2017

  16. [24]

    IEEE T-IP13(4), 600–612 (2004)

    Zhou Wang, Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE T-IP13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    2004

  17. [25]

    In: CVPR

    Zhu, Z., Peng, S., Larsson, V., Xu, W., Bao, H., Cui, Z., Oswald, M.R., Pollefeys, M.: Nice-slam: Neural implicit scalable encoding for slam. In: CVPR. pp. 12786– 12796 (2022)

    2022

  18. [26]

    In: ICCV

    Zoran, D., Isola, P., Krishnan, D., Freeman, W.T.: Learning ordinal relationships for mid-level vision. In: ICCV. pp. 388–396 (2015)

    2015

  19. [27]

    In: ECCV (2018)

    Zou, Y., Luo, Z., Huang, J.B.: Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In: ECCV (2018)

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.