{"id":"017b3f5e-78d1-4fd8-b40a-fdef1d9cbed6","arxiv_id":"2504.17719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DSPPs achieve the best in-distribution calibration, but deep ensembles are more robust to synthetic feature shifts than either deep GP method.","lead":"This paper compares how deep Gaussian processes, deep sigma point processes, and deep ensembles estimate uncertainty on two tabular datasets, both in normal conditions and when the input features are artificially corrupted. It reports that sigma point processes give the best in-distribution calibration, while deep ensembles remain the most robust under synthetic feature shifts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (12) defines DSPP predictions as a Gaussian mixture, but Eqs. (19) and (22) score a single Gaussian; the mixture-to-moment collapse is never stated, leaving the headline calibration numbers undefined.","rationale":"The reader's weakest_assumption identifies exactly this gap: the DSPP mixture is defined in Eq. (12), but the regression metrics in Eqs. (19) and (22) assume a single Gaussian, with no stated collapse. This concern is the most load-bearing because every quantitative statement about in-distribution calibration — the ECE of 0.026 vs. 0.112 and the NLL of 2.985 vs. 3.387 — depends on which predictive object is being scored. The paper presents the comparison clearly and includes code, which is credit in its favor, but the missing definition prevents the reader from verifying the central quantitative claim from the equations alone. I do not think this warrants rejection: classification metrics and the no-hidden-layer DSPP regression setup partly mitigate the issue, and it is directly fixable by stating the evaluation protocol and re-running the numbers. The lack of error bars and the synthetic nature of the shifts are secondary concerns; the mixture ambiguity is more fundamental because it affects the meaning of the headline metrics. Hence the verdict remains CONDITIONAL, not ACCEPT or REJECT.","tokens_in":13249,"tokens_out":10293,"duration_ms":115950,"concrete_test":"Use the released repository (github.com/matthjs/xai-gp) and recompute the CASP test metrics for DSPP and DGP in two ways: (i) exact mixture NLL, −log Σ_j ω_j N(y; μ_j(x), σ_y²), with ECE computed from the mixture CDF; and (ii) moment-matched Gaussian NLL/ECE as implied by Eqs. (19) and (22). Compare both to Table 2, and inspect the evaluation code to determine which definition was actually used. If the two definitions differ materially (for example, DSPP ECE shifts by more than 0.02, or the DSPP/ensemble NLL ordering changes), the headline calibration claim must be restated or revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Eq. (12) defines the DSPP predictive distribution as a Q-component Gaussian mixture: p_dspp(y_i|x_i) = Σ_j ω^(j) p^(j)(y_i|x_i). Yet the regression metrics in §3.1 are written for a single Gaussian: Eq. (19) computes NLL as ½log(2πσ²(x)) + (y−μ(x))²/(2σ²(x)), and Eq. (22) builds prediction intervals from one μ and one σ. The paper never states whether μ and σ² are the moments of the mixture, whether the exact mixture density is used for NLL, or whether mixture quantiles are used for interval-based ECE. This is not a cosmetic gap: NLL is a proper scoring rule only for the exact predictive distribution, and interval calibration depends on the whole CDF, not just the first two moments. If the evaluation collapses the mixture to moments, then the reported DSPP NLL and ECE score a Gaussian approximation rather than the trained model; if it does not, then Eqs. (19) and (22) are not the formulas used. Because the central in-distribution claim rests on Table 2 (DSPP ECE 0.026 vs. deep ensemble 0.112 on CASP), this ambiguity is load-bearing. Classification is less vulnerable because p_dspp is evaluated directly as a softmax mixture, and the CASP DSPP has no hidden layers, so its exact predictive is near-Gaussian. But the same ambiguity affects DGP, whose predictive is an MC mixture, and any deeper DSPP regression model; the model-agnostic formulas in §3.1 are insufficient to reproduce the numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper empirically compares Deep Gaussian Processes (DGPs), Deep Sigma Point Processes (DSPPs), and Deep Ensembles on two tabular benchmarks: regression on the CASP protein structure dataset and binary classification on the ESR seizure recognition dataset. It reports MAE/accuracy, NLL, and ECE for in-distribution performance, and evaluates ECE/MAE/accuracy under five synthetic feature-level perturbations at increasing severity. The authors conclude that DSPPs provide the strongest in-distribution calibration (ECE 0.026 on CASP, 0.035 on ESR), while Deep Ensembles are the most robust under shift, maintaining stable ECE and accuracy while GP-based methods degrade. The paper includes ablations on inducing points and depth, and makes code publicly available.","tokens_in":13638,"tokens_out":7494,"duration_ms":65884,"significance":"If the central results survive verification, this is a useful benchmark study: it extends the evaluation of DSPPs beyond NLL, applies DSPPs to classification with a softmax likelihood, and applies a uniform shift protocol across regression and classification. The provision of code, public datasets, and ablations strengthens reproducibility. However, the paper's quantitative claims are currently undermined by an unstated transformation of the mixture predictive distributions into the single-Gaussian NLL/interval formulas, by a likely misspecified ECE formula, and by the absence of error bars or statistical tests for the headline comparisons. These issues affect the central claims and require correction before the results can be interpreted.","major_comments":[{"comment":"The DSPP predictive is defined in Eq. (12) as a Q-component Gaussian mixture, and the DGP predictive is a Monte Carlo mixture, but the regression metrics in §3.1 are written for a single Gaussian: Eq. (19) evaluates NLL with one μ(x) and σ²(x), and Eq. (22) builds prediction intervals from the same two moments. The paper never states whether μ and σ² are the moments of the mixture, whether the exact mixture density is used for NLL, or whether interval calibration uses mixture quantiles. This is load-bearing because NLL is a proper scoring rule only for the exact predictive distribution, and interval-based ECE depends on the full CDF; if the mixture is collapsed to moments, the headline DSPP calibration numbers in Table 2 score a Gaussian approximation rather than the trained model, and if it is not, Eqs. (19) and (22) do not describe the evaluation. Please specify the exact evaluation procedure and recompute the affected numbers if needed.","section":"§2.3, §3.1, Eqs. (12), (19), (22)"},{"comment":"The ECE formula in Eq. (21) uses |B_b|/B as the bin weight instead of |B_b|/N, even though the bins partition the N test points and Σ_b |B_b| = N. With this formula the reported ECE values are not on the standard probability scale and are not comparable to the usual ECE definition; please correct the denominator and recompute Table 2 and Figures 4/7 if the implementation follows Eq. (21).","section":"§3.1, Eq. (21)"},{"comment":"The paper does not report any measure of variability for Table 2, despite §3.5 stating that N = 5 independent training runs were performed per model for the shift experiments; it is unclear whether the Table 2 entries are single-seed results, averages, or best runs. The shift results are presented only as boxplots aggregating over runs and corruption types, with no confidence intervals, paired tests, or effect-size measures. As a consequence, the headline contrasts (DSPP ECE 0.026 vs. Deep Ensemble 0.112 on CASP; ensemble stability near 0.11 ECE under shift vs. GP degradation) cannot be distinguished from run-to-run variation. Please report means and standard deviations or CIs over the runs and add a statistical comparison (e.g., paired by seed and corruption type) for the shift outcomes.","section":"§3.5, Table 2, Figures 4 and 7"},{"comment":"The shift analysis pools all five corruption types into a single boxplot per severity level, but the corruptions have qualitatively different effects—feature permutation preserves the marginal distribution of each column, whereas Gaussian noise, scaling, masking, and outliers change marginal or joint structure. Pooling conceals per-corruption differences that are needed to support the conclusion that Deep Ensembles are 'more robust' and GP methods 'vulnerable' in general. Please report per-corruption results (or include corruption type as a factor in the analysis) before drawing general robustness conclusions.","section":"§3.5, Eqs. (24)–(28), Figures 4 and 7"},{"comment":"The optimized DSPP for CASP has architecture [ ] (no hidden layers), so the headline in-distribution calibration advantage on the regression task (ECE 0.026) is achieved by a shallow sparse GP, not by a deep sigma point process. This limits the strength of the claim that 'DSPPs provide strong in-distribution calibration' for deep models on regression; please either include at least one deep DSPP configuration in the main CASP comparison or explicitly temper the depth-related interpretation.","section":"Appendix A, Table 3"}],"minor_comments":[{"comment":"The citation [27] for the statement that NLL can overemphasize tail probabilities is Watkins and Dayan's Q-learning paper; the intended reference is likely [21]. Please correct.","section":"§3.1, reference [27]"},{"comment":"Typo: 'agumented' should be 'augmented'.","section":"§2.3"},{"comment":"Typo: 'This work focused on standard on the standard regression and classification' should read 'standard regression and classification'.","section":"§5.1"},{"comment":"Typo: 'Sezure' should be 'Seizure'.","section":"Figure 3 caption"},{"comment":"The opening sentence, 'For this experiment, we were interested in the relationship between the number of and the negative log likelihood for our dataset,' is missing a word (likely 'layers'). Please fix.","section":"Appendix C.2"},{"comment":"The first displayed line of Eq. (30) is missing the logarithm on q_ψ(z); it should read E_{q_ψ(z)}[log q_ψ(z) − log(p_θ(x|z)p_θ(z)/p_θ(x))].","section":"Appendix B, Eq. (30)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a measurement study and the central ambiguity in metric definition is fixable; I recommend major revision rather than rejection. The lack of error bars is pervasive and should be addressed. Also, the ECE formula in Eq. (21) appears to be an implementation-affecting typo; if the code reproduces it, all ECE values need recomputation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an honest empirical comparison of three UQ approaches, and the first benchmark I know that puts DSPPs through ECE and distribution-shift testing, including a softmax classification setup. The qualitative conclusions look plausible: DSPPs calibrate best in-distribution on these two datasets, deep ensembles hold up better under feature perturbations, and DGPs sit somewhere in between. The paper deserves a serious referee, but right now the headline DSPP numbers are not reproducible as written.\n\nThe main problem is the one the stress-test note flags. Eq. (12) defines the DSPP predictive as a Q-component Gaussian mixture, but Eqs. (19) and (22) write NLL and interval-based ECE for a single Gaussian. The paper never says whether μ and σ are mixture moments, whether the exact mixture density is used for NLL, or whether mixture quantiles are used for the intervals. That ambiguity is load-bearing because the central in-distribution claim (DSPP ECE 0.026 vs. ensemble 0.112 on CASP) depends entirely on that choice. For classification it's less serious because the softmax mixture is scored directly, and the CASP DSPP has no hidden layers so its predictive is near-Gaussian, but the same gap affects any deeper regression DSPP and the DGP, whose predictive is an MC mixture. This needs to be stated and justified.\n\nBeyond that, Table 2 reports single numbers without error bars or significance tests, even though the shift experiment uses 5 runs. The boxplots show distributions but there's no statistical comparison between models. Two datasets is a small basis for the broader claim about GP-based methods; the abstract also overstates the case by saying GP-based methods showed vulnerabilities when DSPP actually had the best ECE under shift on CASP. These are ordinary weaknesses, not fatal ones.\n\nThe good parts: the hyperparameter tuning is done with a proper validation split, the shift framework is clearly specified, the ablations on inducing points and depth are a nice addition, and the code is public. Nothing in the results looks manufactured; the numbers and figures line up.\n\nThe paper is for applied ML people choosing between UQ methods. It fills a real gap and the flaws are addressable with a revision. I would send it to a venue that accepts empirical evaluation papers, with a request for major revision, and would expect the authors to clarify the DSPP metric definitions and add quantitative comparisons.","headline":"First DSPP/ensemble calibration benchmark, but the DSPP scoring is underspecified and the headline numbers need tightening before I'd fully trust them.","tokens_in":14125,"tokens_out":4186,"would_cite":true,"duration_ms":39652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep sigma point processes calibrate best in-distribution, but deep ensembles hold up better under distribution shift.","keywords":["deep Gaussian processes","deep sigma point processes","uncertainty calibration","expected calibration error","distribution shift","deep ensembles","negative log likelihood"],"falsifier":"Recompute DSPP NLL and ECE from the full Gaussian mixture of its predictive distribution instead of a collapsed single Gaussian; if Table 2 changes materially, the claimed calibration advantage is an artifact of the collapse. Independently, run the same models under real covariate shift datasets to see whether the ensemble's stability persists beyond the paper's five synthetic perturbations.","tokens_in":13070,"feed_emoji":"🎯","tokens_out":6539,"duration_ms":60959,"temperature":0.7,"pith_summary":"This paper asks whether hierarchical Gaussian process models deliver what they promise in practice: well-calibrated uncertainty. It compares Deep Gaussian Processes (DGPs) and Deep Sigma Point Processes (DSPPs) against a Deep Ensemble of neural networks on a protein regression task and an EEG seizure classification task. The central finding is a split verdict: DSPPs achieve the lowest in-distribution calibration error on both datasets, while the Deep Ensemble holds its calibration and accuracy steady under synthetic feature perturbations that degrade the GP-based models. The result matters because it separates two goals—calibrated uncertainty on familiar data and safety under distribution shift—and shows they can point to different models.","feed_headline":"GP models calibrate best, but ensembles win under shift","feed_subtitle":"DSPP posts the lowest in-distribution error, but deep ensembles stay accurate when features are corrupted.","key_machinery":"The central object is the Deep Sigma Point Process (DSPP), whose layers propagate uncertainty through learnable quadrature points (sigma points) instead of Monte Carlo sampling; this produces a finite Gaussian mixture predictive distribution (Eq 12) and enables direct maximum-likelihood training. The evaluation machinery consists of NLL and ECE metrics plus five synthetic feature-level perturbations (Gaussian noise, feature masking, feature scaling, feature permutation, outlier injection) applied at increasing severity.","core_discovery":"On the paper's own terms, the discovery is that DSPPs' sigma-point approximation yields excellent in-distribution calibration (ECE 0.026 on CASP, 0.035 on ESR) but that this advantage does not carry over to robustness: under the five feature-level shifts, the Deep Ensemble maintains roughly stable ECE and accuracy, whereas DGP and DSPP degrade—DGP sharply on regression. The paper concludes that good in-distribution calibration does not guarantee robustness, and that deep ensembles remain a strong baseline for deployment under shift.","pith_inferences":["If the goal is deployment under shift, an immediate next test is whether the ranking holds under label-preserving covariate shifts that change feature marginal distributions in ways the five synthetic perturbations do not.","The paper leaves implicit how the DSPP mixture predictive is reduced to the single mean and variance used in NLL and interval metrics; spelling out that reduction, or evaluating the mixture directly, would let others reproduce the calibration numbers.","The ensemble's stability suggests diversity across members, rather than Bayesian posterior propagation, protects calibration under feature corruption; comparing DSPPs with more sigma points would test whether the gap is architectural or inferential."],"forward_implications":["On tabular regression and classification, a DSPP is the better choice when in-distribution calibration is the priority.","For deployment where input features may be corrupted, a deep ensemble is the safer default because it held ECE near 0.11 on CASP and accuracy near 0.97 on ESR while the GP models degraded.","NLL alone is not enough to judge uncertainty quality; the severity curves of ECE reveal robustness differences that a single NLL average hides.","DSPP's sigma-point training works with a softmax likelihood, so its calibration benefit is not limited to Gaussian regression."],"supporting_citations":[{"why":"Introduces deep Gaussian processes, the architecture being evaluated.","marker":"[6]"},{"why":"Defines deep sigma point processes and the sigma-point approximation that yields the calibration advantage.","marker":"[14]"},{"why":"Defines deep ensembles, the baseline that shows robustness under shift.","marker":"[16]"},{"why":"Provides the distribution-shift evaluation methodology that the paper adapts.","marker":"[20]"},{"why":"Supplies the CASP protein regression dataset used in the regression experiments.","marker":"[22]"},{"why":"Supplies the ESR epileptic seizure recognition dataset used in the classification experiments.","marker":"[1]"}],"fun_headline_variants":["DSPP calibrates best, but ensembles endure shift","Deep GP calibration shines, robustness fades under shift","Ensembles outperform deep GPs when data shifts","Sigma-point GP calib strong, but shift exposes fragility","In-distribution DSPP wins, out-of-distribution ensembles win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper evaluates DSPPs with a single mean and variance while its equations define a mixture predictive distribution; that unstated collapse is load-bearing for every calibration number reported.","fun_headline_variants_meta":{"raw":{"variants":["DSPP calibrates best, but ensembles endure shift","Deep GP calibration shines, robustness fades under shift","Ensembles outperform deep GPs when data shifts","Sigma-point GP calib strong, but shift exposes fragility","In-distribution DSPP wins, out-of-distribution ensembles win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1624,"prompt_tokens":855,"completion_tokens":769,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":689}},"tokens_in":471,"tokens_out":769,"duration_ms":6465,"temperature":1.0,"reasoning_tokens":689,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:32:05.655975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute DSPP NLL and ECE from the full Gaussian mixture of its predictive distribution instead of a collapsed single Gaussian; if Table 2 changes materially, the claimed calibration advantage is an artifact of the collapse. Independently, run the same models under real covariate shift datasets to see whether the ensemble's stability persists beyond the paper's five synthetic perturbations.","supporting_citations":[{"cited_title":"Simple and scal- able predictive uncertainty estimation using deep ensembles","cited_arxiv_id":null,"evidence_quote":"Defines deep ensembles, the baseline that shows robustness under shift."},{"cited_title":"Lawrence","cited_arxiv_id":null,"evidence_quote":"Introduces deep Gaussian processes, the architecture being evaluated."},{"cited_title":"Deepsigmapointprocesses","cited_arxiv_id":null,"evidence_quote":"Defines deep sigma point processes and the sigma-point approximation that yields the calibration advantage."},{"cited_title":"Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state","cited_arxiv_id":null,"evidence_quote":"Supplies the ESR epileptic seizure recognition dataset used in the classification experiments."}],"review_version":1}