Pith. sign in

REVIEW 3 major objections 5 minor 181 references

This thesis claims that a model's training trajectory—the sequence of intermediate checkpoints from a single run—carries an uncertainty signal strong enough for state-of-the-art selective prediction, and that the gap to an ideal abstaining

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A training-dynamics abstention method matches deep ensembles at a fraction of the training cost, and a five-term error budget explains why selective classifiers still fall short of the oracle.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid thesis compilation: SPTD is empirically useful but the central premise is conditional on SGD noise, and the gap decomposition is diagnostic rather than a proven theorem. the 3 major comments →

arxiv 2508.07556 v2 pith:TBCUVOS4 submitted 2025-08-11 cs.LG cs.AIcs.CYstat.ML

Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning

Stephan Rabanser This is my paper
classification cs.LG cs.AIcs.CYstat.ML
keywords selective predictionuncertainty quantificationtraining dynamicscheckpoint ensemblesdifferential privacyselective classification gapcalibrationverifiable inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis tries to establish that a model's own training history is a rich source of uncertainty that can be exploited without changing its architecture or loss. By weighting disagreements between intermediate checkpoints and the final model, the proposed SPTD method achieves selective prediction (abstaining on uncertain inputs) on par with deep ensembles while using a single training run, and it remains valid under differential privacy because it only post-processes saved checkpoints. The thesis then asks why real selectors fall short of the perfect-ranking oracle, and proves a finite-sample decomposition of that gap into Bayes noise, approximation error, ranking error, statistical noise, and a residual term for optimization and distribution shift. A key consequence is that monotone post-hoc calibration cannot close the gap because it preserves score ordering; progress requires methods that re-rank. The same uncertainty machinery is shown to be manipulable by adversaries, motivating verifiable-confidence defenses.

Core claim

The central claim is that uncertainty can be effectively inferred from the inherent dynamics of the training process—specifically, from the instability of a test point's prediction across intermediate checkpoints—rather than retrofitted through costly modifications to the model or its training procedure. Concretely, the thesis proposes a weighted prediction-instability score g(x) = Σ_t (t/T)^k a_t(x), where a_t marks disagreement between checkpoint t and the final model, and shows that thresholding this score yields state-of-the-art selective classification, selective regression, and selective time-series forecasting, while remaining fully compatible with differential privacy. The thesis fur

What carries the argument

The machinery that carries the argument is the checkpointed training trajectory itself. From a single SGD run, the method computes a weighted disagreement score between intermediate models and the final model, using the weighting (t/T)^k to emphasize late-training instability; under the SGD-as-tempered-Bayesian-sampler view, this score acts as a proxy for posterior predictive variance. The second half of the thesis is carried by an error-budget identity: the selective-classification gap Δ(c) ≤ ε_Bayes(c)+ε_approx(c)+ε_rank(c)+ε_stat(c)+ε_misc(c), each term tied to a measurable, addressable source of looseness.

Load-bearing premise

The load-bearing premise is that a test point whose prediction keeps changing between late-training checkpoints is genuinely more likely to be misclassified by the final model; if the training run converges to a single smooth basin where checkpoints rarely disagree, the disagreement signal carries no information and the method loses its stopping rule.

What would settle it

Train a network on a fixed dataset using full-batch gradient descent (no mini-batch noise) so that late checkpoints agree almost perfectly on all test points; if the resulting disagreement score is near-zero everywhere and the accuracy–coverage curve is flat (selective accuracy at 50% coverage equals full-coverage accuracy), the central claim that checkpoint instability encodes misclassification risk is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, any deployed model whose checkpoints were saved during training can be given a high-quality abstention rule with no retraining, no architecture changes, and no loss-function modifications.
  • Training-dynamics abstention is compatible with differential privacy at no additional privacy cost, since checkpoints are post-processed outputs of the private training algorithm; this makes selective prediction usable in privacy-sensitive deployments.
  • The gap decomposition gives practitioners an error budget: to approach oracle performance at a given coverage, spend effort on capacity, labels, validation data, robust training, or ranking-aware scoring rather than defaulting to calibration.
  • Calibration alone cannot fix ranking errors; interventions that re-order confidence scores (e.g., feature-aware or non-monotone calibrators, ensemble-based scores) are the only route to closing the ranking term.
  • Since the same instability signal can be forged by an adversary, deployed uncertainty estimates need integrity guarantees; calibration audits plus verifiable inference are proposed as a defense.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the thesis leaves implicit: deliberately raising gradient noise during training (e.g., larger mini-batch variance) should sharpen the disagreement signal for hard examples, while near-deterministic optimizers should weaken it.
  • Because the weighted instability score is computed from saved checkpoints, already-trained models with recorded training telemetry could gain a reliability layer for free—turning historical training logs into an abstention mechanism.
  • The five-term decomposition could be turned into an operational gap-attribution tool: given a deployed selector and a labeled validation set, a practitioner could estimate each ε term and decide which intervention is most cost-effective.
  • The adversarial manipulation result suggests that any uncertainty-based deferral policy is gameable when the model operator is untrusted; the verifiable-inference defense could extend to other trust metrics such as fairness or robustness certificates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This PhD-thesis manuscript argues that uncertainty can be read off from the training trajectory rather than from architectural or loss modifications. It proposes SPTD/SCTD: for a test point, compute a weighted sum of disagreements between intermediate checkpoints and the final model, and abstain when this instability score is high. The manuscript reports strong selective-prediction results on CIFAR-10/100, StanfordCars, Food101, several regression datasets, and M4/Hospital time series; analyzes selective classification under DP-SGD, concluding that SPTD is the most robust choice and introducing an accuracy-normalized evaluation score; derives a five-term finite-sample decomposition of the selective-classification gap; and studies adversarial manipulation of uncertainty. The central thesis statement claims that training-dynamics uncertainty is a more direct and broadly applicable path to robust selective prediction.

Significance. If the core premise holds, SPTD is practically valuable: it requires no architecture or loss changes, applies across classification, regression, and forecasting, is cheaper to train than deep ensembles, and preserves DP guarantees by post-processing. The DP study is well targeted, and the proposed accuracy-normalized evaluation score is a useful tool for comparing selectors across privacy levels. The gap decomposition, if made fully formal, would provide a valuable error budget for selective-classification research. Strengths include extensive evaluation with repeated seeds and detailed hyperparameters, a released codebase for the DP study, and clear algorithms for the proposed method. The main weaknesses are that the generality of the central premise is not tested outside noisy SGD and that the main text does not make the formal content of the gap-decomposition theorem checkable.

major comments (3)
  1. [§3.4.3 / §3.4.5, §1.2] The load-bearing premise of the thesis is Eq. (3.12), which equates checkpoint disagreement to posterior predictive variance under the SGD-as-sampler view. If training converges to a single basin, g(x) can be near zero for both correct and incorrect points, and the signal vanishes. The manuscript acknowledges this in §3.4.5 ('diminished model diversity') but never tests it. All experiments use noisy SGD, so the broad claim that uncertainty is 'inherent' to training dynamics is not established for stable-convergence regimes such as full-batch GD, large-batch SGD, SWA, or aggressive LR decay. Please add experiments in at least one such regime and, if the signal degrades, restrict the thesis statement to the stochastic-optimization setting.
  2. [§5.3.3, Eq. (5.1), Thm 5.3.4] The claimed first finite-sample decomposition is central, but the main text gives only qualitative descriptions of the five terms; the proof is deferred to Appendix C.2.1 and was not available in the review material. As written, if ε_misc is defined as the residual after subtracting the other four terms, Eq. (5.1) is an identity and the 'miscellaneous' term cannot be interpreted as an identified source. Also, the oracle bound acc(afull,c) in Eq. (5.5) is derived under 'no Bayes noise', yet ε_Bayes appears as a separate term; the relationship between these two objects needs to be stated explicitly. Please provide formal definitions, estimators, and coverage of all terms, and clarify what 'finite-sample' means (N-dependence and probability).
  3. [Abstract, §3.1, Table 3.2] The 'new SOTA' claim for SPTD is stronger than the data. In Table 3.2, SPTD alone is often statistically tied with or slightly worse than DE: e.g., CIFAR-100 at 80% coverage DE is 84.7 vs SPTD 84.6; at 60% DE is 90.2 vs SPTD 90.1; CIFAR-10 at 60% DE is 99.6 vs SPTD 99.6. Only DE+SPTD is uniformly best. Please attribute the SOTA claim to the combined method or temper the wording for SPTD alone.
minor comments (5)
  1. [§3.3.2 vs §5.2] The gating score sign is inconsistent: in Algorithms 1–3 low g means accept, while in Definition 5.2.1 high g means accept (g ≥ τ). Please use a single convention or explicitly state the mapping between the two.
  2. [§3.3.3] The assumption that expected disagreement e_t ≈ 0 throughout training is used to derive sMAX and sSUM but is only verified empirically later. Label it as an empirical assumption rather than a formal premise.
  3. [Table 3.1] In Table 3.1, SPTD training space is listed O(1) while inference space is O(T), which conflates checkpoint storage with training space. Please clarify how checkpoint storage is accounted for.
  4. [§4.3.4, Eq. (4.6)] The accuracy-normalized score is written as an integral but the approximation uses a sum over coverage levels. Define the coverage grid and the threshold-selection procedure.
  5. [§3.2, Eq. (3.8) and Table 4.2] The MSIS formula has malformed parentheses and an unclear normalization factor; also, Table 4.2 uses NNTD while the text uses SCTD for the same method. Please correct these presentation issues.

Circularity Check

0 steps flagged

No significant circularity: SPTD is an empirical method validated against external baselines, and the theoretical justifications are assumptions rather than derived predictions.

full rationale

The thesis is a compilation of the author's own papers, so self-citations are pervasive, but they are not used to substitute for evidence. Chapter 3 introduces SPTD, whose score g(x) is defined directly from checkpoint disagreement with the final model; the underlying premise that late-training instability correlates with error is presented as a hypothesis ('We hypothesize that such training time correlations with uncertainty also hold for test points') and then tested empirically against external baselines (SR, SAT, DE) on standard benchmarks (CIFAR, Food101, StanfordCars, etc.). The SGD-as-tempered-Bayesian view (Eq. 3.12) is explicitly framed as a view, not a theorem, and the paper acknowledges failure modes ('diminished model diversity') in §3.4.5. No fitted parameter is renamed as a prediction: the weighting k is a hyperparameter analyzed by ablation, not fit to test data. Chapter 4 studies SPTD under DP, comparing to external SC methods, and proposes a new evaluation metric; the upper bound in Eq. 4.5 is a simple mathematical identity derived from full-coverage accuracy, and the contribution is the metric itself, not a hidden prediction. Chapter 5's gap decomposition (Eq. 5.1) is an inequality with a residual 'miscellaneous' term; while such a residual can in principle absorb slack, the paper provides independent theoretical results (e.g., monotone calibration cannot re-rank) and controlled empirical isolation of components. The integral of the gap is explicitly acknowledged to be equivalent to the existing E-AURC metric (Geifman et al., 2018), so there is no renaming of a known result as novel. Overall, the central claim that training dynamics provide a cheap, post-hoc abstention signal is supported by external comparisons and is not equivalent to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

Core assumptions: SPTD relies on a heuristic coupling between SGD dynamics and posterior uncertainty; the DP analysis relies on standard DP composition; the gap decomposition is a claimed theorem whose proof was not fully inspected here.

free parameters (5)
  • k (checkpoint weighting exponent) = k in [1,3] selected empirically
    Controls how strongly late-training disagreements are penalized in SPTD; Figure 3.6 shows k in [1,3] performs best, suggesting tuning on the benchmark.
  • number of checkpoints T = typically 25-50 (subsampling to 10 works at high coverage)
    Checkpointing resolution is a design choice; full trajectory gives best low-coverage performance.
  • selection threshold tau = varies by coverage level, set on validation
    Standard threshold for accept/reject in selective prediction; not a scientific constant but a fitted operating point.
  • Mirage loss tradeoff lambda = lambda tradeoff introduced in Appendix D.3.1
    Balances attack utility against inducing uncertainty; likely tuned per dataset.
  • DP-SGD noise multiplier and clipping norm = noise multiplier chosen adaptively to meet epsilon; clipping norm c=10
    Privacy mechanism parameters, not fitted to improve selective prediction, but they affect uncertainty quality.
axioms (5)
  • standard math Chebyshev inequality bounds the probability that a datapoint was in the training set (Lemma 3.3.1)
    Used to motivate the forging-based derivation of SPTD scores.
  • domain assumption SGD iterates sample from a tempered Bayesian posterior (Mandt et al., 2017; Zhang et al., 2019)
    Section 3.4.1 and Eq. 3.12: checkpoint disagreement is interpreted as posterior predictive variance; this is the theoretical bridge from optimization noise to uncertainty.
  • ad hoc to paper The expected disagreement e_t is approximately 0 throughout training
    Section 3.3.3: 'by assuming e_t=0 always' simplifies the Chebyshev bound into the max and sum scores; the text says removing the assumption gives similar performance.
  • standard math Differential privacy post-processing and composition properties (Dwork et al., 2006)
    Section 4.3.2 classifies SC methods by whether they are post-processing, direct optimization, or sequential composition; this is standard DP theory.
  • domain assumption Five-way additive decomposition of the selective gap is valid with non-overlapping terms (Eq. 5.1)
    The finite-sample decomposition is the chapter's theorem; its validity as an inequality with interpretable terms is assumed/incomplete in the truncated text.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning." pith.science (2026). https://pith.science/paper/TBCUVOS4

@misc{pith2026250807556,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBCUVOS4}},
  note         = {Machine review of arXiv:2508.07556}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Machine learning (ML) systems are increasingly deployed in high-stakes domains where reliability is paramount. This thesis investigates how uncertainty estimation can enhance the safety and trustworthiness of ML, focusing on selective prediction -- where models abstain when confidence is low. We first show that a model's training trajectory contains rich uncertainty signals that can be exploited without altering its architecture or loss. By ensembling predictions from intermediate checkpoints, we propose a lightweight, post-hoc abstention method that works across tasks, avoids the cost of deep ensembles, and achieves state-of-the-art selective prediction performance. Crucially, this approach is fully compatible with differential privacy (DP), allowing us to study how privacy noise affects uncertainty quality. We find that while many methods degrade under DP, our trajectory-based approach remains robust, and we introduce a framework for isolating the privacy-uncertainty trade-off. Next, we then develop a finite-sample decomposition of the selective classification gap -- the deviation from the oracle accuracy-coverage curve -- identifying five interpretable error sources and clarifying which interventions can close the gap. This explains why calibration alone cannot fix ranking errors, motivating methods that improve uncertainty ordering. Finally, we show that uncertainty signals can be adversarially manipulated to hide errors or deny service while maintaining high accuracy, and we design defenses combining calibration audits with verifiable inference. Together, these contributions advance reliable ML by improving, evaluating, and safeguarding uncertainty estimation, enabling models that not only make accurate predictions -- but also know when to say "I do not know".

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

181 extracted references · 40 canonical work pages · 7 internal anchors

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    B., Mironov, I., Talwar, K., and Zhang, L

    Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.\ 308--318, 2016

    2016

  3. [3]

    and Kang, D

    Adila, D. and Kang, D. Understanding out-of-distribution: A perspective of data dynamics. In I (Still) Can't Believe It's Not Better! Workshop at NeurIPS 2021, pp.\ 1--8. PMLR, 2022

    2021

  4. [4]

    Estimating example difficulty using variance of gradients

    Agarwal, C., D'souza, D., and Hooker, S. Estimating example difficulty using variance of gradients. arXiv preprint arXiv:2008.11600, 2020

    Pith/arXiv arXiv 2008

  5. [5]

    C., Rangapuram, S., Salinas, D., Schulz, J., et al

    Alexandrov, A., Benidis, K., Bohlke-Schneider, M., Flunkert, V., Gasthaus, J., Januschowski, T., Maddix, D. C., Rangapuram, S., Salinas, D., Schulz, J., et al. Gluonts: Probabilistic time series models in python. arXiv preprint arXiv:1906.05264, 2019

    Pith/arXiv arXiv 1906

  6. [6]

    Concrete problems in ai safety

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

    Pith/arXiv arXiv 2016

  7. [7]

    Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    Baek, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35: 0 19274--19289, 2022

    2022

  9. [9]

    Differential privacy has disparate impact on model accuracy

    Bagdasaryan, E., Poursaeed, O., and Shmatikov, V. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32, 2019

    2019

  10. [10]

    Deep learning through the lens of example difficulty

    Baldock, R., Maennel, H., and Neyshabur, B. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34, 2021

    2021

  11. [11]

    QSAR fish toxicity

    Ballabio, D., Cassotti, M., Consonni, V., and Todeschini, R. QSAR fish toxicity . UCI Machine Learning Repository, 2019. DOI : https://doi.org/10.24432/C5JG7B

    work page doi:10.24432/c5jg7b 2019

  12. [12]

    Window-based distribution shift detection for deep neural networks

    Bar-Shalom, G., Geifman, Y., and El-Yaniv, R. Window-based distribution shift detection for deep neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  13. [13]

    Bartlett, P. L. and Wegkamp, M. H. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9 0 (8), 2008

    2008

  14. [14]

    Private empirical risk minimization: Efficient algorithms and tight error bounds

    Bassily, R., Smith, A., and Thakurta, A. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pp.\ 464--473. IEEE, 2014

    2014

  15. [15]

    Stability of stochastic gradient descent on nonsmooth convex losses

    Bassily, R., Feldman, V., Guzm \'a n, C., and Talwar, K. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33: 0 4381--4391, 2020

    2020

  16. [16]

    and Kohavi, R

    Becker, B. and Kohavi, R. Adult . UCI Machine Learning Repository, 1996. DOI : https://doi.org/10.24432/C5XW20

    doi:10.24432/c5xw20 1996

  17. [17]

    Berger, J. O. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013

    2013

  18. [18]

    Bishop, C. M. Pattern recognition and machine learning. Springer, 2: 0 1122--1128, 2006

    2006

  19. [19]

    T., and Nakkiran, P

    B asiok, J., Gopalan, P., Hu, L., Kalai, A. T., and Nakkiran, P. Loss Minimization Yields Multicalibration for Large Neural Networks . In Guruswami, V. (ed.), 15th Innovations in Theoretical Computer Science Conference (ITCS 2024), volume 287 of Leibniz International Proceedings in Informatics (LIPIcs), pp.\ 17:1--17:21, Dagstuhl, Germany, 2024. Schloss D...

    work page doi:10.4230/lipics.itcs.2024.17 2024

  20. [20]

    M., Kucukelbir, A., and McAuliffe, J

    Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: A review for statisticians. Journal of the American statistical Association, 112 0 (518): 0 859--877, 2017

    2017

  21. [21]

    Weight uncertainty in neural network

    Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference on machine learning, pp.\ 1613--1622. PMLR, 2015

    2015

  22. [22]

    Food-101 -- mining discriminative components with random forests

    Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 -- mining discriminative components with random forests. In European Conference on Computer Vision, 2014

    2014

  23. [23]

    and Honaker, J

    Brawner, T. and Honaker, J. Bootstrap inference and differential privacy: Standard errors for free. Unpublished Manuscript, 2018

    2018

  24. [24]

    Bagging predictors

    Breiman, L. Bagging predictors. Machine learning, 24: 0 123--140, 1996

    1996

  25. [25]

    Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78 0 (1): 0 1--3, 1950

    1950

  26. [26]

    Universally composable security: a new paradigm for cryptographic protocols

    Canetti, R. Universally composable security: a new paradigm for cryptographic protocols. In Proceedings 42nd IEEE Symposium on Foundations of Computer Science, pp.\ 136--145, 2001. doi:10.1109/SFCS.2001.959888

    work page arXiv 2001

  27. [27]

    Cattelan, L. F. P. and Silva, D. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. 2023 a

    2023

  28. [28]

    Cattelan, L. F. P. and Silva, D. Improving selective classification performance of deep neural networks through post-hoc logit normalization and temperature scaling. arXiv preprint arXiv:2305.15508, 2023 b

    Pith/arXiv arXiv 2023

  29. [29]

    Artificial intelligence, bias and clinical safety

    Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., and Tsaneva-Atanasova, K. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28 0 (3): 0 231--237, 2019

    2019

  30. [30]

    Active bias: Training more accurate neural networks by emphasizing high variance samples

    Chang, H.-S., Learned-Miller, E., and McCallum, A. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30, 2017

    2017

  31. [31]

    Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019

    2019

  32. [32]

    Checkpoint ensembles: Ensemble methods from a single training process

    Chen, H., Lundberg, S., and Lee, S.-I. Checkpoint ensembles: Ensemble methods from a single training process. arXiv preprint arXiv:1710.03282, 2017

    Pith/arXiv arXiv 2017

  33. [33]

    An optimum character recognition system using decision functions

    Chow, C.-K. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, 0 (4): 0 247--254, 1957

    1957

  34. [34]

    Coenen, L., Abdullah, A. K. A., and Guns, T. Probability of default estimation, with a reject option. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp.\ 439--448, 2020. doi:10.1109/DSAA49011.2020.00058

    arXiv 2020

  35. [35]

    Learning with rejection

    Cortes, C., DeSalvo, G., and Mohri, M. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp.\ 67--82. Springer, 2016

    2016

  36. [36]

    Unbiased statistical estimation and valid confidence intervals under differential privacy

    Covington, C., He, X., Honaker, J., and Kamath, G. Unbiased statistical estimation and valid confidence intervals under differential privacy. arXiv preprint arXiv:2110.14465, 2021

    Pith/arXiv arXiv 2021

  37. [37]

    Multiparty computation from somewhat homomorphic encryption

    Damg rd, I., Pastro, V., Smart, N., and Zakarias, S. Multiparty computation from somewhat homomorphic encryption. In Safavi-Naini, R. and Canetti, R. (eds.), Advances in Cryptology -- CRYPTO 2012, pp.\ 643--662, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642-32009-5

    2012

  38. [38]

    A probabilistic theory of pattern recognition, volume 31

    Devroye, L., Gy \"o rfi, L., and Lugosi, G. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013

    2013

  39. [39]

    Retiring adult: New datasets for fair machine learning

    Ding, F., Hardt, M., Miller, J., and Schmidt, L. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34: 0 6478--6490, 2021

    2021

  40. [40]

    Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off

    Ding, Y., Liu, J., Xiong, J., and Shi, Y. Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.\ 4--5, 2020

    2020

  41. [41]

    Calibrating noise to sensitivity in private data analysis

    Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.\ 265--284. Springer, 2006

    2006

  42. [42]

    The algorithmic foundations of differential privacy

    Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9 0 (3--4): 0 211--407, 2014

    2014

  43. [43]

    $p$-DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations

    Dziedzic, A., Rabanser, S., Yaghini, M., Ale, A., Erdogdu, M. A., and Papernot, N. p -DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations . arXiv preprint arXiv:2207.12545, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  44. [44]

    and Wiener, Y

    El-Yaniv, R. and Wiener, Y. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 0 (53): 0 1605--1641, 2010. URL http://jmlr.org/papers/v11/el-yaniv10a.html

    2010

  45. [45]

    El-Yaniv, R. et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 0 (5), 2010

    2010

  46. [46]

    Statistically valid inferences from privacy protected data

    Evans, G., King, G., Schwenzfeier, M., and Thakurta, A. Statistically valid inferences from privacy protected data. American Political Science Review, 2019

    2019

  47. [47]

    and Zhang, C

    Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 0 2881--2891, 2020

    2020

  48. [48]

    Selective prediction-set models with coverage rate guarantees

    Feng, J., Sondhi, A., Perry, J., and Simon, N. Selective prediction-set models with coverage rate guarantees. Biometrics, 2021

    2021

  49. [49]

    O., Hajimirsadeghi, H., and Abdi, A

    Feng, L., Ahmed, M. O., Hajimirsadeghi, H., and Abdi, A. H. Towards better selective classification. In The Eleventh International Conference on Learning Representations, 2023

    2023

  50. [50]

    Parametric bootstrap for differentially private confidence intervals

    Ferrando, C., Wang, S., and Sheldon, D. Parametric bootstrap for differentially private confidence intervals. In International Conference on Artificial Intelligence and Statistics, pp.\ 1598--1618. PMLR, 2022

    2022

  51. [51]

    Conformal prediction: a unified review of theory and new challenges

    Fontana, M., Zeni, G., and Vantini, S. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29 0 (1): 0 1--23, 2023

    2023

  52. [52]

    A., Thomas, M

    Franzese, N., Dziedzic, A., Choquette-Choo, C. A., Thomas, M. R., Kaleem, M. A., Rabanser, S., Fang, C., Jha, S., Papernot, N., and Wang, X. Robust and Actively Secure Serverless Collaborative Learning . Advances in Neural Information Processing Systems, 36: 0 39504--39528, 2023

    2023

  53. [53]

    Constant-overhead zero-knowledge for RAM programs

    Franzese, O., Katz, J., Lu, S., Ostrovsky, R., Wang, X., and Weng, C. Constant-overhead zero-knowledge for RAM programs. Cryptology ePrint Archive, Paper 2021/979, 2021. URL https://eprint.iacr.org/2021/979

    2021

  54. [54]

    and Ghahramani, Z

    Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp.\ 1050--1059. PMLR, 2016

    2016

  55. [55]

    Gal, Y. et al. Uncertainty in deep learning. 2016

    2016

  56. [56]

    What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers

    Galil, I., Dabbah, M., and El-Yaniv, R. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers. arXiv preprint arXiv:2302.11874, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  57. [57]

    Selective classification via one-sided prediction

    Gangrade, A., Kag, A., and Saligrama, V. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics, pp.\ 2179--2187. PMLR, 2021

    2021

  58. [58]

    Experimenting with zero-knowledge proofs of training

    Garg, S., Goel, A., Jha, S., Mahloujifar, S., Mahmoody, M., Policharla, G.-V., and Wang, M. Experimenting with zero-knowledge proofs of training. Cryptology ePrint Archive, Paper 2023/1345, 2023. URL https://eprint.iacr.org/2023/1345

    2023

  59. [59]

    Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56 0 (Suppl 1): 0 1513--1589, 2023

    2023

  60. [60]

    and El-Yaniv, R

    Geifman, Y. and El-Yaniv, R. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017

    2017

  61. [61]

    and El-Yaniv, R

    Geifman, Y. and El-Yaniv, R. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning, pp.\ 2151--2159. PMLR, 2019

    2019

  62. [62]

    Bias-reduced uncertainty estimation for deep neural classifiers

    Geifman, Y., Uziel, G., and El-Yaniv, R. Bias-reduced uncertainty estimation for deep neural classifiers. arXiv preprint arXiv:1805.08206, 2018

    Pith/arXiv arXiv 2018

  63. [63]

    Geyer, C. J. Practical markov chain monte carlo. Statistical science, pp.\ 473--483, 1992

    1992

  64. [64]

    Ghodsi, Z., Hari, S. K. S., Frosio, I., Tsai, T., Troccoli, A., Keckler, S. W., Garg, S., and Anandkumar, A. Generating and characterizing scenarios for safety testing of autonomous vehicles. arXiv preprint arXiv:2103.07403, 2021

    Pith/arXiv arXiv 2021

  65. [65]

    and Raftery, A

    Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102 0 (477): 0 359--378, 2007

    2007

  66. [66]

    The knowledge complexity of interactive proof-systems

    Goldwasser, S., Micali, S., and Rackoff, C. The knowledge complexity of interactive proof-systems. In Proceedings of the Seventeenth Annual ACM Symposium on Theory of Computing, STOC '85, pp.\ 291–304, New York, NY, USA, 1985. Association for Computing Machinery. ISBN 0897911512. doi:10.1145/22145.22178. URL https://doi.org/10.1145/22145.22178

    arXiv 1985

  67. [67]

    When does a predictor know its own loss?

    Gollakota, A., Gopalan, P., Karan, A., Peale, C., and Wieder, U. When does a predictor know its own loss? arXiv preprint arXiv:2502.20375, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  68. [68]

    Bounded-abstaining classification for breast tumors in imbalanced ultrasound images

    Guan, H., Zhang, Y., Cheng, H.-D., and Tang, X. Bounded-abstaining classification for breast tumors in imbalanced ultrasound images. International Journal of Applied Mathematics and Computer Science, 30 0 (2), 2020

    2020

  69. [69]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning, pp.\ 1321--1330. PMLR, 2017

    2017

  70. [70]

    Scalable zero-knowledge proofs for non-linear functions in machine learning

    Hao, M., Chen, H., Li, H., Weng, C., Zhang, Y., Yang, H., and Zhang, T. Scalable zero-knowledge proofs for non-linear functions in machine learning. In 33rd USENIX Security Symposium (USENIX Security 24), pp.\ 3819--3836, Philadelphia, PA, August 2024. USENIX Association. ISBN 978-1-939133-44-1. URL https://www.usenix.org/conference/usenixsecurity24/prese...

    2024

  71. [71]

    Equality of opportunity in supervised learning

    Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016 a

    2016

  72. [72]

    Train faster, generalize better: Stability of stochastic gradient descent

    Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp.\ 1225--1234. PMLR, 2016 b

    2016

  73. [73]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  74. [74]

    Multicalibration: Calibration for the (computationally-identifiable) masses

    H \'e bert-Johnson, U., Kim, M., Reingold, O., and Rothblum, G. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning, pp.\ 1939--1948. PMLR, 2018

    1939

  75. [75]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019

    2019

  76. [76]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016

    Pith/arXiv arXiv 2016

  77. [77]

    Scaling out-of-distribution detection for real-world settings

    Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019

    Pith/arXiv arXiv 1911

  78. [78]

    Statlog (German Credit Data)

    Hofmann, H. Statlog (German Credit Data) . UCI Machine Learning Repository, 1994. DOI : https://doi.org/10.24432/C5NC77

    doi:10.24432/c5nc77 1994

  79. [79]

    What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05248, 2019

    Hooker, S., Courville, A., Clark, G., Dauphin, Y., and Frome, A. What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05248, 2019

    Pith/arXiv arXiv 1911

  80. [80]

    Detection of traffic signs in real-world images: The G erman T raffic S ign D etection B enchmark

    Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., and Igel, C. Detection of traffic signs in real-world images: The G erman T raffic S ign D etection B enchmark. In International Joint Conference on Neural Networks, number 1288, 2013

    2013

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.