{"id":"e040f64d-92a6-4a55-a38d-67f4eb0747d4","arxiv_id":"1909.01193","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A self-supervised deep network denoises consumer depth maps by using multi-view color consistency as the training signal, removing the need for ground truth depth.","lead":"This paper trains a deep network to clean noisy depth maps from consumer cameras without needing perfect depth data as an answer key. Instead, it uses several camera views of the same scene and checks that colors match when projected through the predicted clean depth.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative claim rests on untreated K2-as-ground-truth proxy; reported margins are within plausible cross-sensor alignment noise, so outperformance is not yet established without exact-GT validation and significance testing.","rationale":"The reader's weakest_assumption (K2 as close-to-GT proxy) is exactly the load-bearing point. The entire set of quantitative claims, direct depth denoising and KinectFusion, uses K2 depth as ground truth for D415 data. The paper does not quantify the proxy's bias or alignment error, and the margins between the top methods are small (1 mm MAE; 14 mm RMSE). Without exact geometry or statistical testing, the reported ranking could be an artifact of cross-sensor bias and projective misalignment. I agree with the reader's assessment: the method is novel, the self-supervised mechanism is plausible, and the ablation and synthetic results provide partial support, but the central 'outperforms' claim is conditional on validating the evaluation proxy. Therefore the verdict stays CONDITIONAL; the proposed synthetic-GT or laser-scan re-evaluation would settle whether the concern lands. The only additional nuance is that DRR actually beats Ours on all normal-angle thresholds in Table 1, so the conclusion's wording should be metric-qualified even if the depth metrics hold.","tokens_in":16006,"tokens_out":9039,"duration_ms":91212,"concrete_test":"Re-run the Table 1 evaluation on synthetic test frames from InteriorNet with known clean depth and a realistic D415 noise-and-sparsity model, using exact pixel-correspondence ground truth; compute bootstrap 95% confidence intervals for the differences (Ours minus BF) and (Ours minus RGF) on MAE and RMSE. If the confidence intervals include zero or the ranking flips, the K2-proxy results in Table 1 cannot substantiate the outperformance claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the model outperforms both filters and data-driven methods rests on Table 1, where Kinect v2 depth is used as 'close to ground truth' for D415 denoising. This proxy is load-bearing and unvalidated. K2 and D415 have different systematic biases, noise profiles, and resolutions; after stereo calibration and projective association, residual misalignment at depth edges can produce errors of the same order as the reported margins (MAE 25.11 vs 26.11 vs 26.60 mm; RMSE 58.95 vs 73.25 vs 81.35 mm). No error bars, confidence intervals, or significance tests are provided for the 70-pair set, so it is unknown whether the 1 mm MAE advantage over BF or the 14 mm RMSE advantage over RGF is statistically distinguishable from zero. Because the proxy is itself a noisy depth sensor, a method that smooths D415 toward K2-like statistics, or that is accidentally better aligned with K2's bias, can appear superior without being more accurate. The paper's conclusion 'outperformed... in direct depth map denoising' is therefore not yet supported until the same comparison is run against exact geometry. Note also that on normal-angle thresholds in Table 1, DRR is better than Ours on all thresholds, so the 'outperformed' claim is metric-selective and should be qualified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fully convolutional autoencoder for depth-map denoising trained without ground-truth depth. During training, multiple calibrated RGB-D views are used: the network predicts denoised depth for each view, depth-image-based rendering forward-splats colors from source views into each target view, and a photometric loss between the synthesized color and the real target color, combined with depth and surface-normal regularizers, supervises the network. At inference only a single noisy depth map is needed. The authors collect a multi-view Intel RealSense D415 dataset and compare against bilateral, joint bilateral, rolling-guidance filters, DRR, and DDRNet on direct depth metrics, KinectFusion, and Poisson reconstruction, with a supplementary ablation study and InteriorNet experiments.","tokens_in":16280,"tokens_out":4224,"duration_ms":43274,"significance":"If the evaluation were validated, the method would be a practical contribution: it removes the ground-truth bottleneck for depth denoising, is sensor-agnostic in principle, and the ablation gives some evidence that the photometric loss improves on a plain autoencoder. The paper releases code and a new RGB-D corpus, and the self-supervision mechanism, based on forward splatting with depth-confidence weighting, is clearly described. The main risk is the Kinect v2 evaluation proxy, which is load-bearing for the headline claim that the model outperforms both traditional filtering and data-driven methods.","major_comments":[{"comment":"The quantitative comparison to BF, JBF, RGF, DRR, and DDRNet rests entirely on treating Kinect v2 depth as \"close to ground truth\" for D415 data after stereo calibration and projective association. The reported advantages are small (MAE 25.11 mm vs 26.11 mm for BF and 26.60 mm for RGF), and cross-sensor bias, resolution differences, and residual misalignment at depth edges can easily produce errors of the same order. No error bars, confidence intervals, significance tests, or validation of the association accuracy are reported, so the 1 mm MAE margin and even the 14 mm RMSE margin cannot be distinguished from proxy artifacts. The central claim that the model \"outperformed both traditional filtering and data-driven methods\" in direct denoising therefore needs an evaluation against exact geometry or an independently validated target, with uncertainty quantification.","section":"Section 4 (Evaluation Methodology), Table 1"},{"comment":"The normal-angle metrics contradict the unqualified outperformance claim: DRR achieves a lower mean angle error (30.23 vs 32.09 degrees) and higher 10/20/30-degree accuracy at every threshold, and RGF is also better at the 20 and 30 degree thresholds. The paper's own text acknowledges ranking third on surface errors, yet the abstract and conclusion say the model outperformed both traditional filtering and data-driven methods. This claim needs to be qualified to the metrics on which the improvement actually holds, or the normal-angle comparison needs to be addressed with an appropriate aggregate metric.","section":"Section 4 (Results), Table 1"},{"comment":"The ablation does not support the stated conclusion that depth regularization \"aids photometric supervision by constraining it\": the P-only setting has RMSE 58.30 mm versus 58.95 mm for the full P+D+N model, and P-only also has lower or comparable MAE than P+D, while the differences among all ablation rows are only a few millimeters. Because the ablation is used to justify the final loss design and the photometric-supervision contribution, the authors should report repeated runs with variance, significance tests, or at least a metric-by-metric discussion that acknowledges the non-monotonic results.","section":"Supplementary Table 2 (bottom), Section A.5"}],"minor_comments":[{"comment":"The text refers to \"Table ?? (columns 2-7)\" and \"Table ?? (last column)\"; these references should be replaced with the actual table number.","section":"Section 4 (Results)"},{"comment":"The word \"avalable\" in the abstract should be corrected to \"available\".","section":"Abstract"},{"comment":"The notation \"....pt\" for the four neighboring target pixels is confusing and appears malformed; a standard notation such as the four pixel coordinates obtained by flooring and ceiling x and y would be clearer.","section":"Section 3.1, Eq. (2)"},{"comment":"The description of the retrained DDRNet-TC/DDRNet-PC models using \"splatted depth\" as near ground truth should be specified more precisely, including how the splatted depth is computed and how the near-ground-truth quality was assessed.","section":"Supplementary Section A.2"}],"recommendation":"major_revision","confidential_remarks":"The core self-supervised formulation is sound and the paper is worth pursuing, but the quantitative evidence for the central outperformance claim is currently not secure: the Kinect v2 proxy needs an exact-geometry validation, and the normal-angle metrics already show that the stated claim is too strong as written. If the authors can add a validated evaluation and appropriately qualify their claims, the paper would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely self-supervised depth denoiser with a sensible multi-view photometric loss, and the code and data are out. The controlled synthetic evaluation shows real gains over supervised baselines. But the headline real-sensor claim depends on treating Kinect v2 depth as ground truth for D415 data, and that proxy is not validated. The margins over classical filters are small enough that cross-sensor misalignment or bias could explain them. So: novel and worth engaging with, but the paper overstates what is currently proven.\n\nWhat's new: using differentiable forward splatting from multiple views to supervise depth denoising without clean depth. The weighted accumulation and confidence weighting are reasonable, and the ablation in the supplement shows the photometric loss beats a plain autoencoder while the depth and normal regularizers help. The InteriorNet experiment uses exact synthetic GT, and there the method clearly beats DRR and DDRNet (MAE 33.4 vs 86.9 and 140.8). That is real evidence the mechanism works. Code availability is a plus.\n\nWhere it's soft: the main quantitative comparison in Table 1 uses D415 depth denoised by each method, scored against Kinect v2 depth from a different sensor. The two sensors have different bias, noise, and resolution, and after stereo calibration and projective association there will be alignment error at edges. The reported MAE differences between the top methods are about 1 mm, and RMSE differences are roughly 14-25 mm; either could be within the noise of the proxy. No error bars or significance tests are given for the 70-pair set, so I cannot tell if \"best\" means anything statistically. Also, the \"outperformed\" claim is metric-selective: DRR has better normal-angle accuracy at all thresholds, even though it oversmooths. That should be qualified. A second omission is the lack of comparison to other unsupervised or self-supervised denoisers, so the specific advantage over, say, Noise2Noise adapted to depth is not demonstrated.\n\nNet: the core idea is good, the controlled experiments are encouraging, and the code makes it reproducible. The real-sensor evaluation is not yet convincing enough to support the claim of outperforming filters and supervised methods on D415. If the authors validate against exact geometry (e.g., high-quality 3D scans or realistic simulated noise) and add confidence intervals, this would be a solid paper. As is, I'd accept it for peer review but push for a revised evaluation. I'd also bring it to reading group; it's a useful method to know.","headline":"Genuinely self-supervised depth denoising via multi-view photometric consistency, with strong controlled synthetic results, but the real-sensor outperformance claim rests on an unvalidated cross-sensor proxy and needs exact-GT confirmation.","tokens_in":16871,"tokens_out":2673,"would_cite":true,"duration_ms":27241,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-supervised autoencoder learns to denoise consumer depth maps from raw multi-view captures alone, with no clean depth ever shown, and reports the lowest RMSE and lowest KinectFusion error among filter and learned baselines on…","keywords":["depth denoising","self-supervised learning","multi-view supervision","forward splatting","differentiable rendering","photometric consistency","RGB-D sensors","3D reconstruction"],"falsifier":"Re-run the comparison on a static scene measured with an independent high-accuracy reference that does not involve a second consumer depth sensor, such as a laser-scanned or coded structured-light scan aligned to the D415 frame, and check whether the proposed network still beats bilateral filtering and rolling guidance in RMSE; additionally, plot per-distance error curves to see whether the gain is concentrated at ranges where the Kinect v2's bias differs most from the D415's.","tokens_in":15784,"feed_emoji":"🎯","tokens_out":6367,"duration_ms":59914,"temperature":0.7,"pith_summary":"This paper tries to establish that depth-map denoising for consumer sensors can be learned without any clean depth data. The proposed fully convolutional autoencoder is trained on raw multi-view RGB-D captures: since color is much cleaner than depth, the network's denoised depth predictions are used to forward-splat color from neighboring views into a target view, and the mismatch with the real target color becomes the training signal. Depth and surface-smoothness penalties regularize this photometric loss, and at inference only a single depth map is needed. On Intel RealSense D415 data the model reports the lowest RMSE (58.95 mm) and the lowest KinectFusion point-to-plane error (24.74 mm) against bilateral, joint-bilateral, rolling-guide, DRR, and DDRNet baselines.","feed_headline":"Denoising depth without ground truth, via multi-view splatting","feed_subtitle":"Color from neighboring cameras supervises a network that then cleans depth alone, posting the lowest RMSE on D415.","key_machinery":"The central mechanism is differentiable multi-view forward splatting: each source pixel's color is reprojected through its predicted depth into the target view and spread over the four neighboring target pixels with a bilinear weight, a depth-confidence weight $w_d(D,p)=\\exp(-D(p)/\\sigma_D)$, and a radial-distortion weight $w_r(p)$. Accumulating these contributions from all non-target views and normalizing by the splatted weights produces a synthetic target color image $\\hat{I}_t$, whose pixel-wise comparison with the real target image defines the photometric loss $L_{ph}$. Because splatting blends contributions rather than rasterizing, gradients flow to all contributing depth measurements, which implicitly handles occlusion; the photometric term is regularized by a BerHu depth residual $L_{depth}$ and a normal-smoothness surface loss $L_{surface}$.","core_discovery":"The paper's central claim is that view synthesis can replace ground truth in depth denoising: a single fully convolutional autoencoder, fed only raw depth, learns to suppress sensor noise because its predicted depths must be good enough to reconstruct the scene's observed color from other viewpoints. The authors implement this with multi-view forward splatting rather than inverse warping, so each source pixel contributes color to a target neighborhood with bilinear and confidence weights, gradients flow to every contributing depth measurement, and occlusions need no explicit depth testing. The self-supervised objective combines a photometric loss (a Charbonnier color term plus SSIM with a Tukey penalty) with a BerHu depth-residual regularizer and a surface-normal smoothness prior; partial convolutions handle the roughly 60% zero values of the RealSense high-accuracy profile. On D415 data the method achieves the lowest depth RMSE and the lowest KinectFusion point-to-plane RMSE among all compared methods, transfers qualitatively to Kinect v2 data without retraining, and runs at about 11 ms per frame.","pith_inferences":["The same forward-splatting supervision could be redirected to depth completion or depth super-resolution rather than denoising, since the loss never requires the depth itself to be clean, only the color.","If the Kinect-v2 proxy carries a systematic sensor bias, part of the reported margin may be bias removal rather than noise suppression; evaluating against two unrelated sensors as proxies in the same benchmark would separate those effects.","The hand-crafted confidence weights (exponential in depth and a radial FoV model) are a natural candidate for a learned confidence network, which could adapt the supervision to textureless regions and sensor-specific distortion in one step.","Multi-view splatting supervision assumes photometric consistency, so strong non-Lambertian surfaces or moving content across views would stress it; per-pixel learned weights or a temporal consistency term are testable extensions."],"forward_implications":["Training requires only raw, synchronized multi-view RGB-D captures with known poses, so new sensor setups can be covered without building synthetic clean-depth pairs the way the DRR and DDRNet pipelines do.","Because inference consumes a single depth map and takes roughly 11 ms per frame on a GTX 1080, the denoiser can be embedded in real-time reconstruction or tracking pipelines.","The denoiser transfers to another sensor (Kinect v2) that it was never trained on, while the supervised CNN baselines fail to generalize to D415 data.","Denoised depth improves downstream 3D reconstruction: KinectFusion point-to-plane RMSE drops to 24.74 mm, and Poisson reconstructions preserve finer surface detail than those built from filter outputs."],"supporting_citations":[{"why":"Supplies the differentiable forward-splatting rendering that turns predicted depth into a synthetic target view, which carries the self-supervision.","marker":"[49]"},{"why":"The inverse-warping view-synthesis scheme the authors reject in favor of forward splatting, defining the design contrast.","marker":"[19]"},{"why":"The DRR baseline; also the near-ground-truth synthesized-pair training approach the method avoids.","marker":"[20]"},{"why":"The DDRNet baseline whose supervised raw-clean pairing the method circumvents and which fails to transfer to D415 data.","marker":"[51]"},{"why":"Partial convolutions used in every layer, handling the sparse valid/invalid depth mask.","marker":"[31]"},{"why":"The encoder-decoder with skip connections that the network architecture is modeled on.","marker":"[40]"},{"why":"The BerHu penalty used for the depth-residual regularization term.","marker":"[28]"},{"why":"The bilateral filter, the main classical baseline.","marker":"[48]"},{"why":"The rolling guidance filter baseline that uses color guidance.","marker":"[53]"},{"why":"The KinectFusion application used to measure point-to-plane reconstruction error.","marker":"[36]"}],"fun_headline_variants":["Self-supervised depth denoising via multi-view splatting","Clean depth maps without ground truth using view synthesis","Multi-view splatting enables label-free depth denoising","Depth denoising from color consistency, no ground truth","View synthesis replaces labels for depth denoising"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative evaluation assumes that depth maps from a Kinect v2 placed next to the D415 are a reliable close-to-ground-truth reference, so that projective alignment and the systematic bias between the two sensors are small enough not to decide the reported error differences.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised depth denoising via multi-view splatting","Clean depth maps without ground truth using view synthesis","Multi-view splatting enables label-free depth denoising","Depth denoising from color consistency, no ground truth","View synthesis replaces labels for depth denoising"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1526,"prompt_tokens":1003,"completion_tokens":523,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":443}},"tokens_in":619,"tokens_out":523,"duration_ms":4503,"temperature":1.0,"reasoning_tokens":443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:24:48.241506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on a static scene measured with an independent high-accuracy reference that does not involve a second consumer depth sensor, such as a laser-scanned or coded structured-light scan aligned to the D415 frame, and check whether the proposed network still beats bilateral filtering and rolling guidance in RMSE; additionally, plot per-distance error curves to see whether the gain is concentrated at ranges where the Kinect v2's bias differs most from the D415's.","supporting_citations":[{"cited_title":"Layer-structured 3D scene inference via view synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies the differentiable forward-splatting rendering that turns predicted depth into a synthetic target view, which carries the self-supervision."},{"cited_title":"Spatial transformer networks","cited_arxiv_id":null,"evidence_quote":"The inverse-warping view-synthesis scheme the authors reject in favor of forward splatting, defining the design contrast."},{"cited_title":"Reconstruction-based pair- wise depth dataset for depth image enhancement using CNN","cited_arxiv_id":null,"evidence_quote":"The DRR baseline; also the near-ground-truth synthesized-pair training approach the method avoids."},{"cited_title":"DDRNet: Depth map denoising and refinement for consumer depth cameras using cascaded CNNs","cited_arxiv_id":null,"evidence_quote":"The DDRNet baseline whose supervised raw-clean pairing the method circumvents and which fails to transfer to D415 data."},{"cited_title":"Reda, Kevin J","cited_arxiv_id":null,"evidence_quote":"Partial convolutions used in every layer, handling the sparse valid/invalid depth mask."},{"cited_title":"U- Net: Convolutional networks for biomedical image segmen- tation","cited_arxiv_id":null,"evidence_quote":"The encoder-decoder with skip connections that the network architecture is modeled on."},{"cited_title":"The adaptive BerHu penalty in robust regression","cited_arxiv_id":null,"evidence_quote":"The BerHu penalty used for the depth-residual regularization term."},{"cited_title":"Rolling guidance filter","cited_arxiv_id":null,"evidence_quote":"The rolling guidance filter baseline that uses color guidance."},{"cited_title":"Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J","cited_arxiv_id":null,"evidence_quote":"The KinectFusion application used to measure point-to-plane reconstruction error."}],"review_version":1}