{"id":"5beb3ca9-3ab0-40cc-84ee-146b42b1d499","arxiv_id":"2411.11935","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A product of pairwise Gaussian win probabilities gives a sampling-free, well-calibrated confidence estimate for top-1 classification, validated on LiDAR segmentation at 15-18x lower inference cost.","lead":"The authors replace Monte Carlo sampling with a closed-form lower bound to estimate confidence from Gaussian logit distributions, producing calibrated, slightly underconfident confidence values for LiDAR semantic segmentation. The method runs 15 to 18 times faster than sampling while matching its calibration error, which is directly relevant for real-time autonomous driving perception.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 'negligible difference' between the lower-bound confidence (Eq. 4) and exact confidence (Eq. 2) is never quantified; the product bound can be arbitrarily loose when several classes overlap, so the central approximation claim is unverified.","rationale":"The reader's weakest_assumption focused on the Gaussian logit assumption, which is indeed fundamental. However, the more immediately load-bearing issue is the unquantified tightness of the lower-bound approximation, because the paper's core methodological claim is that the sampling-free bound is nearly equal to the exact Gaussian-model probability. This gap is not a matter of outside consensus; it is an internal check that the paper should have performed and did not. The reader did note the tightness degradation with class count, so there is partial agreement. I do not propose changing the CONDITIONAL verdict because the missing quantification is a reason for requiring additional evidence, not for rejecting the paper outright. If the proposed test reveals large errors, the verdict should move toward REJECT, but that is not known yet. The concern is precise and falsifiable.","tokens_in":13969,"tokens_out":6984,"duration_ms":69759,"concrete_test":"On the full SemanticKITTI validation set (or a representative 10k-point subset), compute for every point both the lower-bound confidence (Eq. 4, closed form) and the exact Monte Carlo confidence (Eq. 2) using a large number of samples (e.g., 10,000) from the predicted Gaussian distributions. Report the mean absolute difference, the 95th percentile of the absolute difference, and the fraction of points where the bound is more than 0.1 below the exact value. Additionally, compute ACE separately for the lower-bound and exact confidences against true accuracy. If the mean absolute difference is small (<0.01) and the ACE values are close, the 'negligible difference' claim holds. If the gap is large on ambiguous points (e.g., points with exact confidence in [0.3, 0.7]), the central claim fails because the bound materially distorts the confidence estimates that the method relies on.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is replacing Monte Carlo confidence estimation with the closed-form lower bound of Eq. 4, P(X1 >= max_{i>=2} Xi) >= product_{i=2}^C Phi_{1,i}. The paper asserts in Section 4.2 that this bound 'exhibits only a negligible difference' to the exact value, but the only evidence is a scatter plot on a subset of SemanticKITTI samples. No quantitative measure (mean/max absolute error, correlation, fraction of points with large error) is reported. This omission is load-bearing because the bound's tightness is not a mathematical guarantee: it follows from Chebyshev's inequality for monotone functions, and the gap E[prod Phi_i(X)] - prod E[Phi_i(X)] can be substantial when the pairwise probabilities Phi_{1,i} are not all close to 1. For a 20-class problem, if five classes each have pairwise probability 0.7 against the winner, the product is 0.7^5 = 0.168, while the exact probability could be substantially larger (all five events are positively correlated through X1). Such loose bounds would place many points in low-confidence bins, inflating ACE and undermining the claimed 'well-calibrated' behavior. The paper's own CIFAR-100 results (Table 3) already show a larger ACE gap between sampling-free (1.80) and logit-sampling (1.16) than on SemanticKITTI, suggesting degradation with class count. Without quantifying the bound's error on the actual LiDAR validation sets, the claim that sampling-free confidence 'closely approximates' the true confidence is not established, and the method's good ACE could be coincidental or driven by compensating overconfidence in the underlying Gaussian model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a sampling-free confidence estimation method for LiDAR semantic segmentation. A network is trained to predict a Gaussian distribution over each class logit, and the confidence of the predicted class is defined as the probability that its logit exceeds all competing logits. Since this probability lacks a closed form, the authors derive a lower bound as the product of pairwise Gaussian CDF comparisons and use this bound as the confidence estimate. They evaluate on SemanticKITTI and nuScenes with SalsaNext and RangeViT, comparing against temperature scaling, logit-sampling, deep ensembles, MC dropout, and evidential deep learning using ACE and reliability diagrams. The paper reports lower ACE than temperature scaling, underconfident behavior, and a 15-18x speedup over logit-sampling.","tokens_in":14274,"tokens_out":7318,"duration_ms":69861,"significance":"If the central approximation claim is supported, the paper makes a practically useful contribution: a closed-form, parameter-free-at-inference confidence estimate for Gaussian-logit classifiers that avoids Monte Carlo sampling. The lower-bound inequality is proven in Appendix A.1 and the proof is sound. The empirical scope is relevant, covering two LiDAR segmentation benchmarks and two backbone architectures, and the combination with epistemic uncertainty methods follows an established direction. However, the paper currently supports its headline claim mainly with a scatter plot, reports no error bars, and leaves the aleatoric-epistemic combination unspecified, so the significance is conditional on substantial revision.","major_comments":[{"comment":"The claim that the lower-bound confidence 'exhibits only a negligible difference' from the exact confidence is not quantified. Figure 1a is a scatter plot on an unspecified subset of SemanticKITTI test samples with SalsaNext only; no mean or maximum absolute error, correlation coefficient, or fraction of points with large deviation is reported. This is load-bearing because Eq. (4) is a mathematical lower bound whose gap can be substantial: E[prod_i Phi_i(X)] - prod_i E[Phi_i(X)] is not guaranteed small when several pairwise probabilities are not close to 1, and the gap can grow with class count. Please report quantitative tightness statistics (e.g., mean/median/max absolute deviation, 95th percentile, fraction of points with deviation > 0.05) over the full validation sets for both SemanticKITTI and nuScenes and both backbones, and ideally compare the ACE of the bound against the ACE of the exact Monte Carlo confidence.","section":"Section 4.2, Eqs. (2) and (4)"},{"comment":"The text states that 'a trend that also holds for SalsaNext, where the same combination yields an ACE of 1.33%' for the sampling-free method with deep ensembles, but Table 1 reports logit-sampling + DE achieving 1.21% ACE on SalsaNext, which is lower than 1.33%. The best-performing configuration on SalsaNext is therefore not the proposed sampling-free method. This contradiction should be corrected, and the comparison should be qualified accordingly.","section":"Section 4.3, Table 1"},{"comment":"All experimental results are single-run point estimates with no error bars, confidence intervals, or multiple seeds. Given the small differences among top configurations (e.g., 1.70% vs 1.83% on RangeViT SemanticKITTI, or 1.33% vs 1.21% on SalsaNext), the claims of superiority or equivalence are not statistically supported. Please add standard deviations over at least three runs or bootstrap confidence intervals for the main ACE and time measurements.","section":"Tables 1 and 3"},{"comment":"The procedure for combining aleatoric and epistemic uncertainty is not specified. The text says 'combining our aleatoric confidence estimation with DE and MC dropout' but does not state whether the Gaussian parameters are averaged over ensemble members, whether logit samples are drawn per member, or whether the final confidences are averaged. Without this information, the headline combined results (e.g., 1.70% ACE on RangeViT) cannot be reproduced or interpreted.","section":"Section 4.3"},{"comment":"The Gaussian assumption on logits is not validated. The exact probability in Eq. (2) and the lower bound in Eq. (4) are only correct if each class logit is exactly Gaussian with the predicted mean and variance. The CIFAR-100 results in Table 3 show a larger degradation for the sampling-free method (ACE 1.80 vs 1.16 for logit-sampling with DE), which is consistent with the bound loosening or the Gaussian assumption failing as the class count grows. Please provide diagnostics such as normality tests on held-out logit samples, variance-calibration checks, or an analysis of how the bound's tightness varies with the number of competing classes.","section":"Section 3.1.1 and Appendix A.3"}],"minor_comments":[{"comment":"The abstract reports ACE values of 1.70% and 1.33% as the results 'using RangeViT and SalsaNext', but these values correspond to the sampling-free method combined with deep ensembles, not to the plain sampling-free approach (which is 2.06% and 2.15% in Table 1). Please clarify this distinction.","section":"Abstract"},{"comment":"The subset of SemanticKITTI test samples used for Figure 1a is not described. Please specify its size and selection procedure so that the scatter plot can be interpreted.","section":"Section 4.2"},{"comment":"The inference-time units are inconsistent: milliseconds for SemanticKITTI and seconds for nuScenes. Please use a single unit or clearly annotate the columns.","section":"Table 1"},{"comment":"In the proof of the lower bound, the functions f and g are introduced abstractly; identifying them explicitly as Phi_b and Phi_c before defining the constant a would make the argument easier to follow.","section":"Section A.1"},{"comment":"The underconfidence property is presented as an empirical observation, but it follows directly from Eq. (4) being a lower bound. The text should acknowledge this mathematical origin when discussing the safety-advantage claim.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the lower-bound derivation is a sound contribution. However, the central 'negligible difference' claim is not quantitatively supported, the best-result statement for SalsaNext is contradicted by the paper's own Table 1, the aleatoric-epistemic combination is unspecified, and no error bars are given. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful paper with a straightforward but valid mathematical idea, and the practical speedup is real. The main weakness is that the paper's central claim—that the lower bound is negligibly different from exact confidence—is never quantified. I'd send it to review with a request for revision.\n\nThe new thing here is the observation that under Gaussian logits, the probability that the predicted class wins can be bounded below by a product of pairwise CDFs. That's a direct application of Chebyshev's integral inequality, and the proof in Appendix A.1 is correct. It's not in Kendall & Gal, who used sampling. Using this bound for confidence estimation is a practical improvement: the paper shows a 15-18x speedup on SemanticKITTI with no loss in segmentation accuracy, and ACE of 1.70% (RangeViT) and 1.33% (SalsaNext) when combined with deep ensembles. Those numbers beat temperature scaling and match or beat logit-sampling. The comparisons are fair, and the training procedure (predicting mean and variance, training with sampled softmax) is clearly described.\n\nWhere it's soft: the claim that the bound 'exhibits only a negligible difference' to exact confidence is supported by nothing more than a scatter plot on a subset of points. No mean or max error, no correlation, no fraction of points with large deviation. This matters because the bound is not guaranteed to be tight; if several classes have overlapping logits with the winner, the product can be much lower than the true probability. The CIFAR-100 results already hint at this: the sampling-free ACE (1.80 with VGG-16) is noticeably worse than logit-sampling (1.16). So the 'negligible difference' claim needs quantitative evidence on the actual validation sets. Also, there are no error bars or multiple seeds anywhere; for a calibration paper that's a real omission. On the timing, I didn't find the inconsistency the reviewer mentioned; the table numbers appear consistent.\n\nThe Gaussian assumption is the key premise. If logits are heavy-tailed or multimodal, the whole computation becomes approximate. The paper acknowledges this only indirectly in the limitations. For the LiDAR application the results are good enough to be useful, but the paper should be more careful about claiming the bound is 'close' without supporting data.\n\nBottom line: the math is sound, the experiments are suggestive but not comprehensive, and the paper would benefit from a serious referee who can push for quantification of the bound's tightness and some uncertainty estimates. I'd send it to review. It's a worthwhile contribution for the autonomous driving calibration community.","headline":"Useful sampling-free confidence bound with a real speedup, but the paper overclaims the bound's tightness without quantifying it.","tokens_in":14854,"tokens_out":4818,"would_cite":false,"duration_ms":42992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A product of pairwise Gaussian CDFs replaces Monte Carlo sampling for calibrated LiDAR confidence, achieving low calibration error with 15–18x faster inference.","keywords":["confidence calibration","aleatoric uncertainty","LiDAR semantic segmentation","sampling-free uncertainty estimation","Gaussian logits","Adaptive Calibration Error","deep ensembles"],"falsifier":"Measure the empirical distribution of logits from a trained model on a held-out set and compare against the fitted Gaussians, for example with a Kolmogorov-Smirnov or chi-square test; if the deviations are large for the classes that compete with the prediction, the lower bound is no longer a true probability bound for the real logits and the ACE gap relative to exact Monte Carlo confidence should widen. A concrete version on SemanticKITTI: for every point, compute both Equation 2 (exact Monte Carlo with many samples) and Equation 4 (the lower bound); the claim predicts the bound never exceeds the exact value and the average gap stays small, so a violation of that ordering or an average gap large enough to shift ACE by several points on a held-out split would falsify the central claim.","tokens_in":13743,"feed_emoji":"🚗","tokens_out":5582,"duration_ms":49562,"temperature":0.7,"pith_summary":"This paper proposes a sampling-free method for computing well-calibrated confidence values in LiDAR semantic segmentation by treating each class logit as a Gaussian with a learned mean and variance. The confidence is the probability that the predicted class's logit exceeds all competing logits, which is approximated by a closed-form lower bound built from pairwise Gaussian CDFs. The authors show that this lower bound closely matches exact Monte Carlo confidence, is systematically slightly underconfident, and yields low Adaptive Calibration Error (ACE) — 1.70% for RangeViT and 1.33% for SalsaNext on SemanticKITTI when combined with deep ensembles — while cutting inference time by 15–18 times relative to logit-sampling. If correct, the result matters because it makes calibrated, conservatively biased uncertainty estimates practical for real-time safety-critical perception in autonomous driving.","feed_headline":"Sampling-free confidences cut LiDAR calibration cost 15x","feed_subtitle":"A closed-form Gaussian lower bound gives calibrated, conservative confidence estimates at real-time speed.","key_machinery":"The central object is the pairwise Gaussian CDF identity $\\Phi_{1,i} = \\Phi(\\mu_1 - \\mu_i \\mid 0,\\, \\sigma_1^2 + \\sigma_i^2)$ and the lower bound of Equation 4: $P(X_1 \\geq \\max_{i\\geq 2} X_i) \\geq \\prod_{i=2}^C \\Phi_{1,i}$. This reduces the multi-class selection probability — an integral over a product of $C-1$ Gaussian CDFs with no closed form — to a product of $C-1$ one-dimensional Gaussian cumulative probabilities, each computable exactly in closed form. The proof mechanism is the positive covariance of strictly monotone functions of a random variable: since $\\Phi_i(X)$ are all increasing functions of $X$, their product's expectation is at least the product of their expectations. The machinery carries the whole argument by replacing Monte Carlo integration with a deterministic, sampling-free estimate that is guaranteed to be conservative.","core_discovery":"On its own terms, the paper establishes that the multi-class confidence integral $P(X_1 \\geq \\max_{i\\geq 2} X_i)$ can be bounded from below by a product of $C-1$ pairwise Gaussian CDF evaluations, $P(X_1 \\geq X_i)$, each with a closed form. Because Gaussian CDFs are monotone and thus positively correlated under the winning logit's distribution, the expectation of their product is at least the product of their expectations; the paper proves this and shows empirically that the bound is nearly tight, with the residual difference manifesting as slight underconfidence. The authors argue this makes aleatoric uncertainty — sensor noise, sparse and distance-dependent LiDAR points, reflectivity variation — explicitly available at inference time without Monte Carlo draws, and that combining this aleatoric confidence with epistemic uncertainty from deep ensembles or MC dropout yields the best-calibrated confidences among the methods compared.","pith_inferences":["The method is not LiDAR-specific: the Gaussian-logit plus pairwise-CDF lower bound applies to any multi-class classifier whose logits can be modeled as Gaussian, so image segmentation, object detection, and other dense prediction tasks could adopt the same machinery.","The lower bound loses tightness as the number of classes grows, as hinted by the larger ACE gap on CIFAR-100; applying the product only over the top-k competing classes and absorbing the rest into a residual term could trade a little conservativeness for tighter calibration in high-class-count settings.","Because the bound is guaranteed conservative, pairing it with an upper bound on the true confidence would yield an interval estimate, giving downstream planners both a pessimistic and a plausible value to act on.","The training-time cost of predicting per-class variance is the method's main overhead; a shared uncertainty head that amortizes variance prediction across classes could reduce that cost while preserving the lower-bound guarantee."],"forward_implications":["The sampling-free bound lets per-pixel confidence be computed in 0.25–0.28 ms for LiDAR range-view images on a single GPU, while logit-sampling adds 6.82G FLOPs and 15–18× more latency.","The method's systematic underconfidence means error-prone points, such as misclassified bikes, poles, and ambiguous sidewalk/street regions, receive higher uncertainty, which is the desirable direction for downstream risk-aware planning.","Combining the aleatoric confidence with epistemic uncertainty from deep ensembles or MC dropout yields ACE of 1.70% on RangeViT and 1.33% on SalsaNext on SemanticKITTI, outperforming temperature scaling and its ensemble variants.","The pipeline transfers to image classification: on CIFAR-100, the sampling-free approach with deep ensembles achieves ACE 1.80%, approaching the sampling baseline's 1.16% while running far faster."],"supporting_citations":[{"why":"Supplies the Gaussian-logit assumption and the logit-sampling baseline that our method replaces.","marker":"[27]"},{"why":"Defines the temperature scaling baseline and the reliability-diagram evaluation used for comparison.","marker":"[1]"},{"why":"Provides the Adaptive Calibration Error (ACE) metric used for all calibration comparisons.","marker":"[14]"},{"why":"Supplies deep ensembles as the epistemic-uncertainty model combined with the proposed aleatoric confidence.","marker":"[6]"},{"why":"Supplies MC dropout as the alternative epistemic-uncertainty model combined with the proposed approach.","marker":"[7]"},{"why":"Provides the SemanticKITTI LiDAR benchmark used for the main segmentation experiments.","marker":"[12]"},{"why":"Provides the RangeViT transformer backbone used in the experiments.","marker":"[44]"},{"why":"Provides the SalsaNext CNN backbone used in the experiments.","marker":"[40]"}],"fun_headline_variants":["Sampling-free confidence cuts LiDAR segmentation cost","Closed-form bound gives real-time calibrated LiDAR confidence","Underconfident but fast: calibrated LiDAR confidence without sampling","Calibrated confidence at 15x speed without sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every class logit is exactly Gaussian with the mean and variance the network predicts; if real logits are heavy-tailed, multimodal, or otherwise non-Gaussian, the confidence estimate and the lower-bound guarantee are no longer exact, and calibration could degrade as the CIFAR-100 results suggest when the class count grows.","fun_headline_variants_meta":{"raw":{"variants":["Sampling-free confidence cuts LiDAR segmentation cost","Closed-form bound gives real-time calibrated LiDAR confidence","Underconfident but fast: calibrated LiDAR confidence without sampling","Calibrated confidence at 15x speed without sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000426,"raw_usage":{"total_tokens":2151,"prompt_tokens":881,"completion_tokens":1270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1205}},"tokens_in":497,"tokens_out":1270,"duration_ms":11074,"temperature":1.0,"reasoning_tokens":1205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:18:00.880130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the empirical distribution of logits from a trained model on a held-out set and compare against the fitted Gaussians, for example with a Kolmogorov-Smirnov or chi-square test; if the deviations are large for the classes that compete with the prediction, the lower bound is no longer a true probability bound for the real logits and the ACE gap relative to exact Monte Carlo confidence should widen. A concrete version on SemanticKITTI: for every point, compute both Equation 2 (exact Monte Carlo with many samples) and Equation 4 (the lower bound); the claim predicts the bound never exceeds the exact value and the average gap stays small, so a violation of that ordering or an average gap large enough to shift ACE by several points on a held-out split would falsify the central claim.","supporting_citations":[{"cited_title":"Nixon, M","cited_arxiv_id":null,"evidence_quote":"Provides the Adaptive Calibration Error (ACE) metric used for all calibration comparisons."},{"cited_title":"Gal and Z","cited_arxiv_id":null,"evidence_quote":"Supplies MC dropout as the alternative epistemic-uncertainty model combined with the proposed approach."},{"cited_title":"Behley, M","cited_arxiv_id":null,"evidence_quote":"Provides the SemanticKITTI LiDAR benchmark used for the main segmentation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RangeViT transformer backbone used in the experiments."},{"cited_title":"Cortinhal, G","cited_arxiv_id":null,"evidence_quote":"Provides the SalsaNext CNN backbone used in the experiments."}],"review_version":1}