REVIEW 6 cited by
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive {\em uncertainty}. Quantifying uncertainty is especially critical in real-world settings, which often involve input distributions that are shifted from the training distribution due to a variety of factors including sample bias and non-stationarity. In such settings, well calibrated uncertainty estimates convey information about when a model's output should (or should not) be trusted. Many probabilistic deep learning methods, including Bayesian-and non-Bayesian methods, have been proposed in the literature for quantifying predictive uncertainty, but to our knowledge there has not previously been a rigorous large-scale empirical comparison of these methods under dataset shift. We present a large-scale benchmark of existing state-of-the-art methods on classification problems and investigate the effect of dataset shift on accuracy and calibration. We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.
Forward citations
Cited by 6 Pith papers
-
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Current LLMs produce consistent but miscalibrated natural-language descriptors of likelihood and uncertainty from probabilistic predictions and are not yet reliable zero-shot risk communicators.
-
Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation
Variance-Gated Ensembles define a margin-variance uncertainty score and a differentiable normalization layer that suppress high-variance ensemble predictions at linear cost.
-
Condition-Stratified Robustness Analysis of Post-Hoc Calibration Methods for Probabilistic Classifiers
In-dataset robustness of temperature scaling versus isotonic regression is condition-dependent and metric-specific, with TEMP more stable on Brier score and calibration slope.
-
PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic
PaTAS propagates Subjective Logic trust opinions through every neuron of a network and updates parameter trust from gradient evidence, yielding per-prediction trust scores intended to flag poisoned or low-reliability inputs.
-
BD at BEA 2025 Shared Task: MPNet Ensembles for Pedagogical Mistake Identification and Localization in AI Tutor Responses
An ensemble of MPNet classifiers trained with grouped cross-validation and class-weighted loss reaches competitive macro-F1 on mistake identification and location in AI tutor responses.
-
Revisiting Bayesian Model Averaging in the Era of Foundation Models
The paper proposes Bayesian model averaging and an entropy-minimizing weight optimizer for ensembling foundation models, reporting accuracy gains over output averaging on image and text classification tasks.
Discussion (0). Sign in to comment.