Pith. sign in

REVIEW 3 major objections 6 minor 39 references

Indoor Depth Completion with Boundary Consistency and Self-Attention

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An end-to-end network with self-attention and boundary consistency completes single-view indoor depth maps with lower error and sharper boundaries than previous methods on Matterport3D.

desk verdict A well-ablated depth completion method with genuine architectural novelty, but its state-of-the-art claim is not yet reproducible because of the unvalidated Zhang reimplementation and the undefined 5% test filter. read the letter →

arxiv 1908.08344 v3 pith:KB4FIYCG submitted 2019-08-22 cs.CV

Yu-Kai Huang , Tsung-Han Wu , Yueh-Cheng Liu , Winston H. Hsu This is my paper · ORCID
classification cs.CV
keywords depthcompletionself-attentiongatedconvolutionboundaryconsistencyocclusionMatterport3DRGB-Dsurfacenormals
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an end-to-end neural network can complete single-view indoor depth maps from an RGB-D image with lower numerical error and sharper object boundaries than previous methods, by combining two ideas: a gated-convolution self-attention mechanism that lets the network focus on useful features at every layer, and a boundary-consistency auxiliary network that forces the completed depth to preserve occlusion edges. This matters because commodity depth cameras leave large missing holes on shiny, transparent, or distant surfaces, and earlier learning-based methods tend to fill holes by copying and interpolating neighboring depth values, which blurs boundaries and loses scene structure. On the Matterport3D dataset, the proposed network reports lower RMSE and mean error than the prior best method, along with higher SSIM (structural similarity index) and better edge quality. A sympathetic reader would take the paper's contribution to be showing that attention over geometric features plus explicit boundary supervision can replace global optimization in depth completion.

What carries the argument

The load-bearing machinery is the gated-convolution self-attention block, defined by $\text{Output} = \text{Feature} \odot \text{Gating}$, where $\text{Gating} = \sigma(\mathrm{SN}(\mathrm{Conv}_g(I)))$ and $\text{Feature} = \varphi(\mathrm{SN}(\mathrm{Conv}_f(I)))$; this lets each convolutional layer dynamically select which spatial and channel features to pass forward. On top of it, a boundary-consistency network, a U-Net, takes the completed depth map and predicts Sobel-derived occlusion boundaries; the resulting boundary loss is backpropagated into the main network, so depth completion is explicitly trained to keep sharp edges. The inputs are RGB, raw depth, and two learned depth representations, surface normals and occlusion boundaries, following the earlier work that defined the task. The total loss combines an L1 term on observed pixels, SSIM, representation losses, and the boundary-consistency loss.

What would settle it

Run the original released model of the prior method on the same Matterport3D test split and score it with RMSE and mean error; if it reaches or beats 1.092 RMSE or 0.342 mean error, the paper's central claim would fail, and the authors' re-implementation should also be compared directly against the original model's outputs to check faithfulness.

Watch

Extended reading notes

Core claim

The central claim is that replacing plain convolutional feature maps with self-attention gating, and adding an auxiliary network that predicts occlusion boundaries (the edges where one surface occludes another) from the completed depth, makes depth completion both numerically more accurate and structurally sharper. On Matterport3D the model reports RMSE 1.092 and mean error 0.342, compared with 1.316 and 0.461 for the previous state-of-the-art method as re-implemented by the authors, and SSIM rises from 0.762 to 0.799. The ablation study attributes the gain to the components: self-attention alone lowers RMSE from 1.262 to 1.095, and adding boundary consistency improves SSIM further while cutting mean error. The paper also argues that, unlike the prior optimization-based method, the whole pipeline is end-to-end trainable and faster at inference.

Load-bearing premise

The paper's claim to outperform the previous best method rests on the authors' own re-implementation of that method, which was rescored with a different error metric and never checked against the original released model; if that re-implementation is weaker than the real method, the reported advantage is not established.

Editorial extensions

If this is right

  • If the central claim holds, depth completion no longer needs sparse Cholesky global optimization: the same or better accuracy comes from an end-to-end network, which is faster at inference and easier to train.
  • Self-attention on convolution features proves useful beyond image inpainting: it can help a depth network recognize geometric meaning, such as walls, corridors, and distant regions, instead of interpolating large holes from nearby pixels.
  • Boundary consistency, enforced by a Sobel-supervised auxiliary network, is a practical way to make predicted depth maps preserve object edges and improve structural metrics such as SSIM.
  • The reported numbers establish a new baseline on Matterport3D for RGB-D depth completion that future single-view methods can be measured against.
  • The ablation study shows the gains are additive: self-attention, SSIM loss, and boundary consistency each contribute, so later work can select components independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same boundary-consistency recipe could sharpen outputs of other dense predictors, such as surface-normal estimation or semantic segmentation, wherever edge fidelity matters; this is a transfer they do not demonstrate.
  • Because the ground truth comes from multi-view reconstruction rather than a commodity sensor, a natural extension is to evaluate on raw Kinect or RealSense holes with real sensor noise, which the paper does not do.
  • Replacing the Sobel-derived boundary target with a learned boundary detector or semantic edge map during training might strengthen or change the observed gains, but the paper does not compare such alternatives.
  • The self-attention maps suggest the learned gates specialize on holes, backgrounds, and object boundaries, raising the question of whether those maps could serve as weak supervision for boundary detection; the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an end-to-end depth completion method for single RGB-D images, evaluated on Matterport3D. The network uses gated convolution with a self-attention mechanism, takes surface normals and occlusion boundaries as additional inputs, and is trained with a boundary-consistency loss that encourages sharp structures by supervising an auxiliary U-Net with Sobel edges of the ground-truth depth. The central claim is state-of-the-art performance: Table 1 reports RMSE 1.092 versus 1.316 for the prior method of Zhang et al. [39]. Internal ablations in Table 2 show consistent gains from self-attention, SSIM loss, and boundary consistency.

Significance. If the result holds, the paper provides a useful architectural contribution: replacing plain FCN regression with gated self-attention and adding a boundary-consistency loss is a plausible and well-ablated recipe, and the authors release code. However, the headline claim of outperforming the previous state of the art rests on an unvalidated reimplementation of Zhang et al. with a changed metric and on an undefined test-set filter, as detailed below. These protocol issues directly affect the paper's strongest claim, so the significance is conditional on resolving them. The internal ablations are a genuine strength and are machine-checkable from the released code.

major comments (3)
  1. [Sec. 4.3, Table 1, footnote 1] The comparison against Zhang et al. [39] uses the authors' own reimplementation rather than the original code or checkpoints, and the footnote states that the original metric was root median square error while Table 1 reports RMSE. No evidence is given that the reimplementation reproduces the original method's behavior. This makes the claimed state-of-the-art margin (1.316 vs. 1.092 RMSE) not attributable to the proposed method. Please validate the reimplementation against the original published numbers or official checkpoints, report the original metric as well as RMSE, or use the official implementation.
  2. [Sec. 4.1] The paper states that after adopting Zhang's training and testing lists the authors 'remove large bias image pairs (about 5%)', yielding 474 test images, but neither the criterion for 'large bias' nor whether removal occurs in training, testing, or both is specified. Filtering the test set changes the benchmark and makes the comparison irreproducible. Please specify the exact filtering rule, release the list of removed image pairs, and report results on the unfiltered test set as a sensitivity check.
  3. [Sec. 4.3] All reported results come from a single run, yet the text repeatedly describes improvements as 'significant'. Without standard deviations, confidence intervals, or at least results from multiple random seeds, the claim of significant gains over competing methods is not statistically supported. Please provide variance estimates for the main comparison and for the ablation table.
minor comments (6)
  1. [Eqs. (1)-(3)] The abbreviation SN is used before it is defined; please spell out spectral normalization at first use.
  2. [Eq. (4)] The total loss is written as L = LSA - lambda_S LS + ...; the minus sign before lambda_S appears to be a typo, and the formula should likely be LSA + lambda_S LS.
  3. [Table 2, Sec. 4.4.1] The row 'W/O SA' is not clearly identified with a baseline model; the text should state explicitly that this row is the ResNet18 FCN baseline also reported as 'FCN' in Table 1.
  4. [Sec. 4.2, Table 1] The threshold notation '1.252' and '1.253' is ambiguous in the text; please write these as 1.25^2 and 1.25^3 (or use a clear subscript) to match standard delta-metric notation.
  5. [Abstract and full text] The GitHub repository URL differs between the abstract (tsunghan-wu/Depth-Completion) and the full text (patrickwu2/Depth-Completion); please use one consistent URL.
  6. [Sec. 3.2] The text says 'The overview pipeline is shown in Figure 1', but Figure 1 shows qualitative examples, whereas Figure 2 contains the architecture; this reference should point to Figure 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the depth-completion pipeline is trained and evaluated against an external benchmark, and the auxiliary losses are multi-task supervision rather than fitted inputs renamed as predictions.

full rationale

The paper's contribution is an end-to-end network trained on Matterport3D with losses defined in Eqs. 4-8, and its state-of-the-art claim is made against external baselines, including Zhang et al. [39]. No prediction reduces by construction to a fitted constant: self-attention is defined operationally by Eqs. 1-3 as gated convolution, and the boundary-consistency loss supervises a second network's boundary prediction against Sobel edges of the same ground-truth depth used for the main depth loss. That is multi-task supervision, not a definitional equivalence between input and output. The choice of surface normals and occlusion boundaries as additional input channels is justified by prior work [39], which is not a self-citation by the present authors. The re-implementation of [39] and the removal of about 5% of image pairs are legitimate external-validity or protocol concerns, but they are not cases where a claimed result is equivalent to its own inputs by construction. No uniqueness theorem, self-citation chain, or ansatz smuggled in via citation is load-bearing. Therefore no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper relies on external pretrained depth representations (surface normals and occlusion boundaries) from Zhang et al., Sobel edge supervision, and Matterport3D reconstructed ground truth. Loss weights are unreported free parameters; no new physical or conceptual entities are introduced.

free parameters (3)
  • λ_S (SSIM loss weight) = not reported
    In Eq. 4, the SSIM loss weight λ_S is not given; its value affects the trade-off between RMSE and structural quality.
  • λ_BC (boundary consistency loss weight) = not reported
    In Eq. 4, the weight for the boundary consistency loss λ_BC is not reported; it controls how strongly boundary sharpness is enforced.
  • λ_N and λ_B (depth representation loss weights) = not reported
    Weights for normal and boundary representation losses in Eq. 4 are not specified.
assumptions (4)
  • domain assumption Sobel edge maps from ground truth depth are a valid proxy for occlusion boundaries.
    Section 3.2 uses Sobel output as the boundary ground truth B0; if Sobel edges are noisy or misaligned with true occlusion boundaries, the boundary consistency loss will be mis-specified.
  • domain assumption Surface normals and occlusion boundaries from Zhang et al.'s depth representation network are accurate and well-aligned with the RGB-D input.
    Section 3.3 feeds these representations into the self-attention network; their quality is taken from the prior work.
  • domain assumption Matterport3D multi-view reconstructed depth is complete and accurate enough to serve as ground truth for single-view depth completion.
    Section 4.1 relies on Matterport3D reconstructed depth; any reconstruction artifact is treated as ground truth.
  • domain assumption Conventional neural network training with observed-pixel-only supervision generalizes to unobserved pixels.
    Loss and metrics in Sec. 3.4 and 4.2 are computed only over observed pixels, following Zhang et al.; this assumes the model will also be correct in missing regions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Indoor Depth Completion with Boundary Consistency and Self-Attention." pith.science (2026). https://pith.science/paper/KB4FIYCG

@misc{pith2026190808344,
  author       = {Pith},
  title        = {Pith review of: Indoor Depth Completion with Boundary Consistency and Self-Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KB4FIYCG}},
  note         = {Machine review of arXiv:1908.08344}
}
read the original abstract

Depth estimation features are helpful for 3D recognition. Commodity-grade depth cameras are able to capture depth and color image in real-time. However, glossy, transparent or distant surface cannot be scanned properly by the sensor. As a result, enhancement and restoration from sensing depth is an important task. Depth completion aims at filling the holes that sensors fail to detect, which is still a complex task for machine to learn. Traditional hand-tuned methods have reached their limits, while neural network based methods tend to copy and interpolate the output from surrounding depth values. This leads to blurred boundaries, and structures of the depth map are lost. Consequently, our main work is to design an end-to-end network improving completion depth maps while maintaining edge clarity. We utilize self-attention mechanism, previously used in image inpainting fields, to extract more useful information in each layer of convolution so that the complete depth map is enhanced. In addition, we propose boundary consistency concept to enhance the depth map quality and structure. Experimental results validate the effectiveness of our self-attention and boundary consistency schema, which outperforms previous state-of-the-art depth completion work on Matterport3D dataset. Our code is publicly available at https://github.com/tsunghan-wu/Depth-Completion.

Figures

Figures reproduced from arXiv: 1908.08344 by the authors.

Figure 1
Figure 1. Our work produces clearer structures and overcomes [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the network architecture. Main improvement of our method are twofold: self-attention mechanism 3.1 (blue dashed [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. In the visualization result, we can see that our method learns better on the geometric meaning. For example, on the first row, our [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: With the help of boundary consistency and SSIM loss, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: In the visualization result, we can see our proposed SA+SSIM+BC (self-attention, SSIM loss and boundary consistency) performs [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of some self-attention maps. Some filters [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages

  1. [39]

    Zhang and T

    Y . Zhang and T. Funkhouser. Deep depth completion of a sin- gle rgb-d image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 175–185, 2018

    work page 2018

  2. [1]

    Bailey and H

    T. Bailey and H. Durrant-Whyte. Simultaneous localization and mapping (slam): Part ii. IEEE robotics & automation magazine, 13(3):108–117, 2006

    work page 2006

  3. [2]

    Camplani and L

    M. Camplani and L. Salgado. Efficient spatio-temporal hole filling strategy for kinect depth maps. In Three-dimensional image processing (3DIP) and applications Ii , volume 8290, page 82900E. International Society for Optics and Photonics, 2012

    work page 2012

  4. [3]

    Chang, A

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y . Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017

    arXiv 2017

  5. [4]

    Doria and R

    D. Doria and R. J. Radke. Filling large holes in lidar data by inpainting depth gradients. In 2012 IEEE Computer Soci- ety Conference on Computer Vision and Pattern Recognition Workshops, pages 65–72. IEEE, 2012

    work page 2012

  6. [5]

    Durrant-Whyte and T

    H. Durrant-Whyte and T. Bailey. Simultaneous localization and mapping: part i. IEEE robotics & automation magazine, 13(2):99–110, 2006

    2006

  7. [6]

    Eigen, C

    D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems , pages 2366–2374, 2014

    work page 2014

  8. [7]

    Engel, T

    J. Engel, T. Sch ¨ops, and D. Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, pages 834–849. Springer, 2014

    work page 2014

Show all 39 references
  1. [8]

    Geiger, P

    A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013

    2013

  2. [9]

    Grossmann

    P. Grossmann. Depth from focus. Pattern recognition letters, 5(1):63–69, 1987

    1987

  3. [10]

    Han, J.-Y

    Y . Han, J.-Y . Lee, and I. So Kweon. High quality shape from a single rgb-d image under uncalibrated natural illumi- nation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1617–1624, 2013

    2013

  4. [11]

    Harrison and P

    A. Harrison and P. Newman. Image and sparse laser fu- sion for dense scene reconstruction. In Field and Service Robotics, pages 219–228. Springer, 2010

    2010

  5. [12]

    Hazirbas, S

    C. Hazirbas, S. G. Soyer, M. C. Staab, L. Leal-Taix ´e, and D. Cremers. Deep depth from focus. In Asian Conference on Computer Vision, pages 525–541. Springer, 2018

    2018

  6. [13]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn- ing for image recognition. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 770–778, 2016

    2016

  7. [14]

    Henry, M

    P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox. Rgb- d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments. The International Journal of Robotics Research, 31(5):647–663, 2012

    2012

  8. [15]

    Hornacek, C

    M. Hornacek, C. Rhemann, M. Gelautz, and C. Rother. Depth super resolution by rigid body self-similarity in 3d. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1123–1130, 2013

    2013

  9. [16]

    Jaritz, R

    M. Jaritz, R. De Charette, E. Wirbel, X. Perrotton, and F. Nashashibi. Sparse and dense data with cnns: Depth com- pletion and semantic segmentation. In 2018 International Conference on 3D Vision (3DV), pages 52–60. IEEE, 2018

    2018

  10. [17]

    J. Kittler. On the accuracy of the sobel edge detector. Image and Vision Computing, 1(1):37–42, 1983

    1983

  11. [18]

    Laina, C

    I. Laina, C. Rupprecht, V . Belagiannis, F. Tombari, and N. Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international conference on 3D vision (3DV), pages 239–248. IEEE, 2016

    2016

  12. [19]

    J.-H. Lee, M. Heo, K.-R. Kim, and C.-S. Kim. Single-image depth estimation based on fourier domain analysis. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 330–339, 2018

    2018

  13. [20]

    G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro. Image inpainting for irregular holes using par- tial convolutions. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 85–100, 2018

    2018

  14. [21]

    Liu and X

    J. Liu and X. Gong. Guided depth enhancement via anisotropic diffusion. In Pacific-Rim Conference on Multi- media, pages 408–417. Springer, 2013

    2013

  15. [22]

    F. Ma, G. V . Cavalheiro, and S. Karaman. Self-supervised sparse-to-dense: self-supervised depth completion from lidar and monocular camera. arXiv preprint arXiv:1807.00275 , 2018

    2018 arXiv

  16. [23]

    Mac Aodha, N

    O. Mac Aodha, N. D. Campbell, A. Nair, and G. J. Bros- tow. Patch based synthesis for single depth image super- resolution. In European conference on computer vision , pages 71–84. Springer, 2012

    2012

  17. [24]

    Mal and S

    F. Mal and S. Karaman. Sparse-to-dense: Depth predic- tion from sparse depth samples and a single image. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2018

    2018

  18. [25]

    Matsuo and Y

    K. Matsuo and Y . Aoki. Depth image enhancement using local tangent plane approximations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 3574–3583, 2015

    2015

  19. [26]

    Mingolla and J

    E. Mingolla and J. T. Todd. Perception of solid shape from shading. Biological cybernetics, 53(3):137–151, 1986

    1986

  20. [27]

    Mingolla and J

    E. Mingolla and J. T. Todd. Perception of solid shape from shading. In Shape from shading, pages 409–441. MIT Press, 1989

    1989

  21. [28]

    Riegler, M

    G. Riegler, M. R ¨uther, and H. Bischof. Atgv-net: Accurate depth super-resolution. In European conference on computer vision, pages 268–284. Springer, 2016

    2016

  22. [29]

    Ronneberger, P

    O. Ronneberger, P. Fischer, and T. Brox. U-net: Convo- lutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234–241. Springer, 2015

    2015

  23. [30]

    Silberman, D

    N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, pages 746–760. Springer, 2012

    2012

  24. [31]

    Strang and K

    G. Strang and K. Aarikka. Introduction to applied mathe- matics, volume 16. Wellesley-Cambridge Press Wellesley, MA, 1986

    1986

  25. [32]

    Suwajanakorn, C

    S. Suwajanakorn, C. Hernandez, and S. M. Seitz. Depth from focus with your mobile phone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 3497–3506, 2015

    2015

  26. [33]

    Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, et al. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing , 13(4):600–612, 2004

    2004

  27. [34]

    J. Xie, R. Girshick, and A. Farhadi. Deep3d: Fully au- tomatic 2d-to-3d video conversion with deep convolutional neural networks. In European Conference on Computer Vi- sion, pages 842–857. Springer, 2016

    2016

  28. [35]

    Q. Yang, R. Yang, J. Davis, and D. Nist ´er. Spatial-depth su- per resolution for range images. In2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2007

    2007

  29. [36]

    J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv:1806.03589, 2018

    2018 arXiv

  30. [37]

    Yu, S.-K

    L.-F. Yu, S.-K. Yeung, Y .-W. Tai, and S. Lin. Shading-based shape refinement of rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1415–1422, 2013

    2013

  31. [38]

    Zhang, P.-S

    R. Zhang, P.-S. Tsai, J. E. Cryer, and M. Shah. Shape-from- shading: a survey. IEEE transactions on pattern analysis and machine intelligence, 21(8):690–706, 1999

    1999

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.