{"id":"a7fb0c89-87f4-46ec-95b2-054854d460e3","arxiv_id":"2501.08188","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning DepthAnythingV2 with a Gaussian negative log-likelihood loss yields the most reliable pixel-wise uncertainty estimates on indoor, street, and object scenes, but it fails on aerial large-depth data.","lead":"This paper tests five established uncertainty quantification methods on the DepthAnythingV2 depth-prediction model across four scene types. It finds that training with a Gaussian negative log-likelihood loss gives the best uncertainty estimates at no extra compute cost, with a notable caveat on aerial imagery.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The blanket GNLL recommendation is undermined by the paper's own UseGeo results and its admitted loss-scaling failure; the claim needs a domain-qualified caveat.","rationale":"The paper is a useful benchmark and the efficiency analysis is credible, but the headline claim is stronger than the evidence. The same data that supports GNLL on NYUv2 and HOPE also shows it failing on UseGeo, and the paper's own text provides a mechanistic explanation (large depth values inflating the Gaussian NLL loss). This is not a disagreement with outside consensus; it is an internal inconsistency between the abstract/conclusion and Section 5.2/Table 5. The reader's weakest assumption already identified the exchangeability of datasets and the UseGeo counterexample; I agree with that assessment and add that the 'predictive performance on par with baseline' part of the claim is also contradicted on Cityscapes and UseGeo. A conditional verdict is appropriate: the paper should either restrict the recommendation to appropriate depth scales or reparameterize GNLL to handle large-depth domains before claiming general promise. The concrete test would resolve whether the failure is a fixable artifact or an intrinsic limitation, and would determine how the caveat should be phrased.","tokens_in":22012,"tokens_out":3718,"duration_ms":36478,"concrete_test":"Re-run the UseGeo GNLL experiments with depth targets normalized (e.g., log-depth or per-image scaling) while keeping all other settings fixed. If p(unc|ina) improves from 0.237 toward MCD's 0.672 and RMSE remains competitive, the UseGeo failure is a loss-scaling artifact of Eq. 3 and the recommendation should be explicitly qualified to depth range; if it does not improve, GNLL's failure on large-depth scenes is intrinsic and the unqualified claim should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract, Section 6) is that GNLL offers reliable uncertainty estimates while maintaining predictive performance and computational efficiency on par with the baseline. The efficiency part is well supported (Table 2). However, the uncertainty and performance parts are internally contradicted by Table 5: on UseGeo, GNLL has the worst p(unc|ina) of all methods (0.237 vs. 0.672 for MCD on ViT-S), and its RMSE degrades sharply at larger encoders (e.g., 7.810 vs. baseline 6.386 on ViT-B). The paper itself acknowledges in Section 5.2: 'This is likely due to the large depth values in UseGeo, which led to much higher absolute GNLL loss values... no hyperparameter adjustments were made to address this, ensuring comparability but potentially hindering GNLL's optimization in this particular case.' That admission directly undercuts the unqualified 'particularly promising' recommendation. The claim treats four heterogeneous datasets as exchangeable evidence, but GNLL is consistently strong only on NYUv2 and HOPE (indoor/object-scale depths) and clearly fails on the large-depth aerial domain. A practitioner in aerial or geospatial applications would be misled by the abstract. The load-bearing weakness is therefore the unsupported generalization from a subset of favorable domains to a blanket methodological recommendation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically compares five uncertainty quantification (UQ) approaches—Learned Confidence (LC), Gaussian Negative Log-Likelihood (GNLL), Monte Carlo Dropout (MCD), Sub-Ensembles (SE), and Test-Time Augmentation (TTA)—integrated into the DepthAnythingV2 foundation model for metric monocular depth estimation. The evaluation spans three encoder sizes (ViT-S/B/L) and four datasets (NYUv2, Cityscapes, UseGeo, HOPE), reporting depth quality metrics, UQ quality metrics (p(acc|cer), p(unc|ina), PAvPU), and efficiency. The central claim is that GNLL is a particularly promising UQ method because it yields reliable uncertainty estimates with no additional training or inference cost, while preserving predictive performance.","tokens_in":22325,"tokens_out":5422,"duration_ms":49202,"significance":"If the central claim holds, the paper has clear practical value: it would let practitioners add pixel-wise heteroscedastic uncertainty to a strong metric-depth backbone at zero marginal inference cost, which matters for robotics and autonomous-driving deployment. The paper's main strength is the breadth of its empirical matrix—five UQ methods × three encoder scales × four domains—together with explicit efficiency measurements and a detailed experimental protocol. The recommendation is derived from measured metrics rather than from a circular derivation. However, the paper's own results on UseGeo substantially qualify the headline claim, and the absence of repeated-run or statistical evidence weakens the ranking claims; the significance therefore depends on the revision being honest about domain dependence and statistical uncertainty.","major_comments":[{"comment":"The headline claim that GNLL 'consistently' delivers reliable uncertainty estimates is contradicted by Table 5: on UseGeo, GNLL has the lowest p(unc|ina) of any method for all three encoders (0.237/0.293/0.294 vs 0.672/0.657/0.669 for MCD) and its RMSE degrades sharply for ViT-B (7.810 vs 6.386 baseline) and ViT-L (7.082 vs 6.173 baseline). Because the abstract and conclusion present GNLL as generally the most promising approach without this caveat, the central claim is an overgeneralization from the favorable indoor and object-scale domains to the aerial large-depth domain. The recommendation should either be domain-qualified or explicitly justify the weighting of the four datasets.","section":"Abstract; Section 6; Table 5"},{"comment":"The paper itself states that GNLL's failure on UseGeo 'is likely due to the large depth values in UseGeo, which led to much higher absolute GNLL loss values' and that no hyperparameter adjustments were made to address this. This admission makes the cross-dataset comparison non-uniform for GNLL: it is not clear whether the poor UseGeo result is an intrinsic property of GNLL or an artifact of an unnormalized loss on a large-depth domain. Since this result is the main counterexample to the paper's central claim, the authors should either test a scale-invariant GNLL variant (e.g., operating in log-depth space or with a normalized variance parameterization) and report whether UseGeo remains a failure case, or explicitly restrict the conclusion to small- and medium-depth ranges.","section":"Section 5.2, UseGeo paragraph"},{"comment":"All uncertainty-quality conclusions are based on single training runs and point estimates of p(acc|cer), p(unc|ina), and PAvPU, with the certain/uncertain split defined by the per-image median uncertainty. The reported differences are often tiny (e.g., PAvPU 0.529 vs 0.533 for GNLL vs MCD on NYUv2 ViT-S), and the metrics are sensitive to the choice of threshold. Without repeated seeds, confidence intervals, or a statistical test, the paper's ranked claims ('consistently outperforms', 'decisively surpasses') are not supported. At minimum, the authors should report variance across seeds or a bootstrap over test images, and temper the ranking language accordingly.","section":"Section 5.1, uncertainty metrics; Tables 3–6"}],"minor_comments":[{"comment":"The AbsRel value 0.631 for TTA appears to be a typo (likely 0.063, since the neighboring values are 0.074 and 0.063); please correct.","section":"Table 5, UseGeo ViT-B TTA row"},{"comment":"The δ1 value is printed as '0488' rather than '0.488'; please fix the missing decimal point.","section":"Table 4, UseGeo ViT-L MCD row"},{"comment":"The δ3 value is printed as '0945' rather than '0.945'; please fix the missing decimal point.","section":"Table 6, HOPE ViT-S LC row"},{"comment":"Typo: 'the the mean and variance' should read 'the mean and variance'.","section":"Section 4.6"},{"comment":"The attribution to Wang et al. (2024) for adapting Learned Confidence to regression is not self-explanatory, since the cited paper (DUSt3R) is primarily a geometric vision method; please state explicitly which component is being reused.","section":"Section 4.2"},{"comment":"The header 'PAvPU↑' is missing the space used in the other tables ('PAvPU ↑'); please unify the formatting.","section":"Table 4 header"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is a competent empirical study and the experimental matrix is useful, but the abstract and conclusion overstate the consistency of GNLL relative to the paper's own Table 5. The revision should require a domain-qualified claim, and ideally add some statistical support for the ranking claims. I would not reject the paper; the central issue is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid benchmark paper that mostly delivers what it promises, with one caveat that the authors themselves half-acknowledge. The new thing here is the evaluation matrix: five established UQ methods—LC, GNLL, MCD, SE, TTA—each fused with DepthAnythingV2, across three encoder sizes and four depth domains (indoor, urban, aerial, object). No single ingredient is new, but the systematic comparison is missing from the literature and is genuinely useful for practitioners picking a default UQ method.\n\nThe efficiency analysis is the cleanest part: GNLL and LC add zero overhead, while MCD and SE cost factor increases in training and inference. The uncertainty metrics are standard and the tables are extensive. Credit also for the honest passage in Section 5.2 explaining why GNLL fails on UseGeo: large depth values inflate the GNLL loss, and no hyperparameter adjustments were made.\n\nThat honesty is also where the paper's central claim collapses. The abstract and conclusion say GNLL is 'particularly promising' for reliable uncertainty, but on UseGeo it is the worst method on p(unc|ina)—0.237 versus 0.672 for MCD on ViT-S—and its RMSE degrades badly at ViT-B and ViT-L. So the recommendation is really domain-dependent: GNLL is the best choice for indoor and object-scale depths (NYUv2, HOPE, arguably Cityscapes), but a poor default for aerial/large-depth scenes. The paper should say that, not bury it in a paragraph. The stress-test note gets this right.\n\nOther soft spots are minor. The ranking relies on point estimates with no repeated runs or statistical tests; the per-image median threshold is a reasonable choice but worth sensitivity analysis. The tables contain likely typos (e.g., log10 0.435 for TTA ViT-B on NYUv2, and a missing decimal in the LC HOPE row). No code is released, which limits the benchmark's value.\n\nBottom line: a useful reference for the MDE-UQ community, and a reasonable candidate for peer review after revision. The fix is not hard: qualify the recommendation by domain, fix the typos, and ideally provide code or at least seed-based variance. I'd cite it as a comparison point and bring it to a reading group focused on uncertainty in depth models.\n\nRecommendation: send to peer review with major/minor revisions requested.","headline":"Solid benchmark comparing five UQ methods on DepthAnythingV2, but the GNLL recommendation is overgeneralized and the paper's own UseGeo results contradict it.","tokens_in":22828,"tokens_out":2446,"would_cite":true,"duration_ms":23212,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a metric depth foundation model with the Gaussian Negative Log-Likelihood loss yields per-pixel uncertainty estimates at no extra inference cost, while maintaining depth accuracy on par with the baseline.","keywords":["uncertainty quantification","monocular depth estimation","metric depth","foundation models","Gaussian negative log-likelihood","DepthAnythingV2","deep learning","dense regression"],"falsifier":"Run the same five methods with multiple random seeds on a large-depth aerial or long-range outdoor dataset and report confidence intervals for p(uncertain|inaccurate). If GNLL's score remains near 0.24 while Monte Carlo Dropout's stays near 0.67 under the paper's own thresholds, the general claim that GNLL is the most promising approach would not survive.","tokens_in":21834,"feed_emoji":"📏","tokens_out":9246,"duration_ms":78913,"temperature":0.7,"pith_summary":"Uncertainty quantification has been studied for monocular depth estimation, but not systematically combined with modern depth foundation models. This paper fuses five uncertainty methods with DepthAnythingV2 and tests them on four metric-depth datasets spanning indoor, urban, aerial, and robotic-object scenes. Its central claim is that fine-tuning with the Gaussian Negative Log-Likelihood (GNLL) loss is the most promising fusion: GNLL learns a per-pixel variance as a second output channel, requires no extra forward passes, and matches the baseline in training and inference time. The paper argues this makes uncertainty-aware, explainable metric depth practical for real-world, safety-critical deployment.","feed_headline":"One extra output channel adds reliable depth uncertainty for free","feed_subtitle":"Five uncertainty methods on DepthAnythingV2 show Gaussian-NLL matches baseline speed and accuracy while exposing errors.","key_machinery":"The load-bearing object is the Gaussian Negative Log-Likelihood loss, a heteroscedastic regression objective that makes the network output a mean $\\mu$ and a variance $s^2$ per pixel. The loss $L_{\\mathrm{GNLL}}(\\mu,\\hat y) = \\frac12 [(\\hat y-\\mu)^2/s^2 + \\log s^2]$ trains both outputs jointly, so the variance is learned implicitly from the same forward pass that produces the depth estimate. This gives GNLL its decisive practical advantage: the fine-tuned model needs no sampling and no architectural multiplication, matching the baseline's FPS, parameters, and FLOPs. The experimental machinery also includes the uncertainty metrics from Mukhoti and Gal, namely p(accurate|certain), p(uncertain|inaccurate), and PAvPU, with accuracy defined by the $\\delta_1$ threshold (ratio $< 1.25$) and certainty defined by a per-image median uncertainty threshold.","core_discovery":"The paper claims that GNLL is the best way to add uncertainty to DepthAnythingV2 because it produces calibrated pixel-wise uncertainty estimates with no additional computational overhead. By outputting both a predictive mean $\\mu$ and a variance $s^2$, and minimizing $L_{\\mathrm{GNLL}}(\\mu,\\hat y) = \\frac12 [(\\hat y-\\mu)^2/s^2 + \\log s^2]$, the network learns heteroscedastic uncertainty implicitly from a single forward pass, unlike Monte Carlo Dropout, Sub-Ensembles, and Test-Time Augmentation, which require multiple forward passes. On NYUv2, Cityscapes, and HOPE, GNLL delivers the best or tied-best uncertainty metrics, reaching up to 98.0% p(accurate|certain) and 91.2% p(uncertain|inaccurate) on NYUv2, while depth accuracy remains on par with or better than the baseline. The paper also reports that on the large-depth aerial UseGeo dataset GNLL falls behind Monte Carlo Dropout, an outcome it attributes to the loss's logarithmic term being magnified by large depth values.","pith_inferences":["A testable extension the paper leaves open is predicting inverse depth or normalizing the GNLL variance by scene scale, which might remove the UseGeo failure because the $\\log s^2$ term is what inflates the loss for large depths.","The paper's uncertainty metrics use a per-image median threshold, so the reported rankings depend on image composition; a fixed absolute uncertainty threshold would give a more direct test of calibration and could change the method ordering.","The results are consistent with GNLL capturing aleatoric uncertainty only; combining GNLL with a small ensemble or with Monte Carlo Dropout could separate epistemic from aleatoric components, something the paper does not attempt.","The claimed transfer to semantic segmentation and pose estimation is plausible because GNLL requires only an extra output head, but it is not demonstrated; the same protocol could be run on those tasks to confirm that the learned variance is meaningful outside depth."],"forward_implications":["Fine-tuning DepthAnythingV2 with GNLL gives practitioners a per-pixel uncertainty map for the same inference cost as the unmodified model, so uncertainty-aware metric depth becomes feasible in latency-constrained robotics and driving systems.","GNLL keeps depth accuracy on par with or ahead of the baseline across three encoder sizes on NYUv2, Cityscapes, and HOPE, showing that adding uncertainty does not force a performance trade-off in those domains.","The GNLL extension is architecture-agnostic: it only adds a second output channel to the depth head, so the same recipe can be transferred to other dense-regression foundation models.","The efficiency comparison shows that sampling-based methods cost 3 to 10 times more inference time while GNLL matches baseline FPS, making GNLL the only evaluated method with free uncertainty in deployment.","The UseGeo result cautions that GNLL's learned variance degrades when depth values are very large, so domain-specific handling is required before using it on aerial or long-range scenes."],"supporting_citations":[{"why":"Introduces the Gaussian Negative Log-Likelihood loss that the paper identifies as the most promising uncertainty method.","marker":"Nix and Weigend (1994)"},{"why":"Supplies the DepthAnythingV2 foundation model, its architecture, and the fine-tuning recommendations used for all experiments.","marker":"Yang et al. (2024b)"},{"why":"Provides the NYUv2 indoor dataset used to evaluate depth accuracy and uncertainty quality.","marker":"Silberman et al. (2012)"},{"why":"Provides the Cityscapes outdoor urban dataset used in the multi-domain evaluation.","marker":"Cordts et al. (2016)"},{"why":"Provides the UseGeo aerial dataset on which GNLL's uncertainty calibration is weakest.","marker":"Nex et al. (2024)"},{"why":"Provides the HOPE robotics dataset with synthetic training and real test images.","marker":"Tyree et al. (2022)"},{"why":"Supplies the uncertainty metrics p(accurate|certain), p(uncertain|inaccurate), and PAvPU used to rank the methods.","marker":"Mukhoti and Gal (2018)"},{"why":"Supplies Monte Carlo Dropout, one of the five compared methods and the strongest competitor on the aerial dataset.","marker":"Gal and Ghahramani (2016)"},{"why":"Supplies Sub-Ensembles, the efficient ensemble approximation used as one of the five methods.","marker":"Valdenegro-Toro (2023)"},{"why":"Supplies Test-Time Augmentation, the post-hoc uncertainty method compared in the evaluation.","marker":"Ayhan and Berens (2018)"}],"fun_headline_variants":["Gaussian-NLL adds calibrated depth uncertainty at zero cost","One extra output channel yields depth uncertainty for free","Free depth uncertainty via Gaussian-NLL: no extra compute","Calibrated depth uncertainty from a single forward pass","GNLL: calibrated depth uncertainty without slowing the model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that GNLL is the most promising method rests on letting its strong indoor, urban, and object-scene results outweigh its clearly worse performance on the aerial large-depth dataset, and on trusting single-run uncertainty scores without statistical tests.","fun_headline_variants_meta":{"raw":{"variants":["Gaussian-NLL adds calibrated depth uncertainty at zero cost","One extra output channel yields depth uncertainty for free","Free depth uncertainty via Gaussian-NLL: no extra compute","Calibrated depth uncertainty from a single forward pass","GNLL: calibrated depth uncertainty without slowing the model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3706,"prompt_tokens":985,"completion_tokens":2721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2644}},"tokens_in":601,"tokens_out":2721,"duration_ms":19443,"temperature":1.0,"reasoning_tokens":2644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:28:53.361524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five methods with multiple random seeds on a large-depth aerial or long-range outdoor dataset and report confidence intervals for p(uncertain|inaccurate). If GNLL's score remains near 0.24 while Monte Carlo Dropout's stays near 0.67 under the paper's own thresholds, the general claim that GNLL is the most promising approach would not survive.","supporting_citations":[{"cited_title":"A., Weigend, A","cited_arxiv_id":null,"evidence_quote":"Introduces the Gaussian Negative Log-Likelihood loss that the paper identifies as the most promising uncertainty method."},{"cited_title":"Indoor segmentation and support inference from rgbd images","cited_arxiv_id":null,"evidence_quote":"Provides the NYUv2 indoor dataset used to evaluate depth accuracy and uncertainty quality."},{"cited_title":"UseGeo-A UAV-based multi-sensor dataset for geospatial research","cited_arxiv_id":null,"evidence_quote":"Provides the UseGeo aerial dataset on which GNLL's uncertainty calibration is weakest."},{"cited_title":"6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the HOPE robotics dataset with synthetic training and real test images."},{"cited_title":"Sub-ensembles for fast uncertainty estimation in neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies Sub-Ensembles, the efficient ensemble approximation used as one of the five methods."}],"review_version":1}