Pith. sign in

REVIEW 3 major objections 5 minor 181 references

Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This thesis claims that a model's training trajectory—the sequence of intermediate checkpoints from a single run—carries an uncertainty signal strong enough for state-of-the-art selective prediction, and that the gap to an ideal abstaining

desk verdict Solid thesis compilation: SPTD is empirically useful but the central premise is conditional on SGD noise, and the gap decomposition is diagnostic rather than a proven theorem. read the letter →

arxiv 2508.07556 v2 pith:TBCUVOS4 submitted 2025-08-11 cs.LG cs.AIcs.CYstat.ML

Stephan Rabanser This is my paper
classification cs.LGcs.AIcs.CYstat.ML
keywords selectivepredictionuncertaintyquantificationtrainingdynamicscheckpointensemblesdifferentialprivacyclassificationgapcalibrationverifiableinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis tries to establish that a model's own training history is a rich source of uncertainty that can be exploited without changing its architecture or loss. By weighting disagreements between intermediate checkpoints and the final model, the proposed SPTD method achieves selective prediction (abstaining on uncertain inputs) on par with deep ensembles while using a single training run, and it remains valid under differential privacy because it only post-processes saved checkpoints. The thesis then asks why real selectors fall short of the perfect-ranking oracle, and proves a finite-sample decomposition of that gap into Bayes noise, approximation error, ranking error, statistical noise, and a residual term for optimization and distribution shift. A key consequence is that monotone post-hoc calibration cannot close the gap because it preserves score ordering; progress requires methods that re-rank. The same uncertainty machinery is shown to be manipulable by adversaries, motivating verifiable-confidence defenses.

What carries the argument

The machinery that carries the argument is the checkpointed training trajectory itself. From a single SGD run, the method computes a weighted disagreement score between intermediate models and the final model, using the weighting (t/T)^k to emphasize late-training instability; under the SGD-as-tempered-Bayesian-sampler view, this score acts as a proxy for posterior predictive variance. The second half of the thesis is carried by an error-budget identity: the selective-classification gap Δ(c) ≤ ε_Bayes(c)+ε_approx(c)+ε_rank(c)+ε_stat(c)+ε_misc(c), each term tied to a measurable, addressable source of looseness.

What would settle it

Train a network on a fixed dataset using full-batch gradient descent (no mini-batch noise) so that late checkpoints agree almost perfectly on all test points; if the resulting disagreement score is near-zero everywhere and the accuracy–coverage curve is flat (selective accuracy at 50% coverage equals full-coverage accuracy), the central claim that checkpoint instability encodes misclassification risk is falsified.

Watch

Extended reading notes

Core claim

The central claim is that uncertainty can be effectively inferred from the inherent dynamics of the training process—specifically, from the instability of a test point's prediction across intermediate checkpoints—rather than retrofitted through costly modifications to the model or its training procedure. Concretely, the thesis proposes a weighted prediction-instability score g(x) = Σ_t (t/T)^k a_t(x), where a_t marks disagreement between checkpoint t and the final model, and shows that thresholding this score yields state-of-the-art selective classification, selective regression, and selective time-series forecasting, while remaining fully compatible with differential privacy. The thesis fur

Load-bearing premise

The load-bearing premise is that a test point whose prediction keeps changing between late-training checkpoints is genuinely more likely to be misclassified by the final model; if the training run converges to a single smooth basin where checkpoints rarely disagree, the disagreement signal carries no information and the method loses its stopping rule.

Editorial extensions

If this is right

  • If correct, any deployed model whose checkpoints were saved during training can be given a high-quality abstention rule with no retraining, no architecture changes, and no loss-function modifications.
  • Training-dynamics abstention is compatible with differential privacy at no additional privacy cost, since checkpoints are post-processed outputs of the private training algorithm; this makes selective prediction usable in privacy-sensitive deployments.
  • The gap decomposition gives practitioners an error budget: to approach oracle performance at a given coverage, spend effort on capacity, labels, validation data, robust training, or ranking-aware scoring rather than defaulting to calibration.
  • Calibration alone cannot fix ranking errors; interventions that re-order confidence scores (e.g., feature-aware or non-monotone calibrators, ensemble-based scores) are the only route to closing the ranking term.
  • Since the same instability signal can be forged by an adversary, deployed uncertainty estimates need integrity guarantees; calibration audits plus verifiable inference are proposed as a defense.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the thesis leaves implicit: deliberately raising gradient noise during training (e.g., larger mini-batch variance) should sharpen the disagreement signal for hard examples, while near-deterministic optimizers should weaken it.
  • Because the weighted instability score is computed from saved checkpoints, already-trained models with recorded training telemetry could gain a reliability layer for free—turning historical training logs into an abstention mechanism.
  • The five-term decomposition could be turned into an operational gap-attribution tool: given a deployed selector and a labeled validation set, a practitioner could estimate each ε term and decide which intervention is most cost-effective.
  • The adversarial manipulation result suggests that any uncertainty-based deferral policy is gameable when the model operator is untrusted; the verifiable-inference defense could extend to other trust metrics such as fairness or robustness certificates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This PhD-thesis manuscript argues that uncertainty can be read off from the training trajectory rather than from architectural or loss modifications. It proposes SPTD/SCTD: for a test point, compute a weighted sum of disagreements between intermediate checkpoints and the final model, and abstain when this instability score is high. The manuscript reports strong selective-prediction results on CIFAR-10/100, StanfordCars, Food101, several regression datasets, and M4/Hospital time series; analyzes selective classification under DP-SGD, concluding that SPTD is the most robust choice and introducing an accuracy-normalized evaluation score; derives a five-term finite-sample decomposition of the selective-classification gap; and studies adversarial manipulation of uncertainty. The central thesis statement claims that training-dynamics uncertainty is a more direct and broadly applicable path to robust selective prediction.

Significance. If the core premise holds, SPTD is practically valuable: it requires no architecture or loss changes, applies across classification, regression, and forecasting, is cheaper to train than deep ensembles, and preserves DP guarantees by post-processing. The DP study is well targeted, and the proposed accuracy-normalized evaluation score is a useful tool for comparing selectors across privacy levels. The gap decomposition, if made fully formal, would provide a valuable error budget for selective-classification research. Strengths include extensive evaluation with repeated seeds and detailed hyperparameters, a released codebase for the DP study, and clear algorithms for the proposed method. The main weaknesses are that the generality of the central premise is not tested outside noisy SGD and that the main text does not make the formal content of the gap-decomposition theorem checkable.

major comments (3)
  1. [§3.4.3 / §3.4.5, §1.2] The load-bearing premise of the thesis is Eq. (3.12), which equates checkpoint disagreement to posterior predictive variance under the SGD-as-sampler view. If training converges to a single basin, g(x) can be near zero for both correct and incorrect points, and the signal vanishes. The manuscript acknowledges this in §3.4.5 ('diminished model diversity') but never tests it. All experiments use noisy SGD, so the broad claim that uncertainty is 'inherent' to training dynamics is not established for stable-convergence regimes such as full-batch GD, large-batch SGD, SWA, or aggressive LR decay. Please add experiments in at least one such regime and, if the signal degrades, restrict the thesis statement to the stochastic-optimization setting.
  2. [§5.3.3, Eq. (5.1), Thm 5.3.4] The claimed first finite-sample decomposition is central, but the main text gives only qualitative descriptions of the five terms; the proof is deferred to Appendix C.2.1 and was not available in the review material. As written, if ε_misc is defined as the residual after subtracting the other four terms, Eq. (5.1) is an identity and the 'miscellaneous' term cannot be interpreted as an identified source. Also, the oracle bound acc(afull,c) in Eq. (5.5) is derived under 'no Bayes noise', yet ε_Bayes appears as a separate term; the relationship between these two objects needs to be stated explicitly. Please provide formal definitions, estimators, and coverage of all terms, and clarify what 'finite-sample' means (N-dependence and probability).
  3. [Abstract, §3.1, Table 3.2] The 'new SOTA' claim for SPTD is stronger than the data. In Table 3.2, SPTD alone is often statistically tied with or slightly worse than DE: e.g., CIFAR-100 at 80% coverage DE is 84.7 vs SPTD 84.6; at 60% DE is 90.2 vs SPTD 90.1; CIFAR-10 at 60% DE is 99.6 vs SPTD 99.6. Only DE+SPTD is uniformly best. Please attribute the SOTA claim to the combined method or temper the wording for SPTD alone.
minor comments (5)
  1. [§3.3.2 vs §5.2] The gating score sign is inconsistent: in Algorithms 1–3 low g means accept, while in Definition 5.2.1 high g means accept (g ≥ τ). Please use a single convention or explicitly state the mapping between the two.
  2. [§3.3.3] The assumption that expected disagreement e_t ≈ 0 throughout training is used to derive sMAX and sSUM but is only verified empirically later. Label it as an empirical assumption rather than a formal premise.
  3. [Table 3.1] In Table 3.1, SPTD training space is listed O(1) while inference space is O(T), which conflates checkpoint storage with training space. Please clarify how checkpoint storage is accounted for.
  4. [§4.3.4, Eq. (4.6)] The accuracy-normalized score is written as an integral but the approximation uses a sum over coverage levels. Define the coverage grid and the threshold-selection procedure.
  5. [§3.2, Eq. (3.8) and Table 4.2] The MSIS formula has malformed parentheses and an unclear normalization factor; also, Table 4.2 uses NNTD while the text uses SCTD for the same method. Please correct these presentation issues.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SPTD is an empirical method validated against external baselines, and the theoretical justifications are assumptions rather than derived predictions.

full rationale

The thesis is a compilation of the author's own papers, so self-citations are pervasive, but they are not used to substitute for evidence. Chapter 3 introduces SPTD, whose score g(x) is defined directly from checkpoint disagreement with the final model; the underlying premise that late-training instability correlates with error is presented as a hypothesis ('We hypothesize that such training time correlations with uncertainty also hold for test points') and then tested empirically against external baselines (SR, SAT, DE) on standard benchmarks (CIFAR, Food101, StanfordCars, etc.). The SGD-as-tempered-Bayesian view (Eq. 3.12) is explicitly framed as a view, not a theorem, and the paper acknowledges failure modes ('diminished model diversity') in §3.4.5. No fitted parameter is renamed as a prediction: the weighting k is a hyperparameter analyzed by ablation, not fit to test data. Chapter 4 studies SPTD under DP, comparing to external SC methods, and proposes a new evaluation metric; the upper bound in Eq. 4.5 is a simple mathematical identity derived from full-coverage accuracy, and the contribution is the metric itself, not a hidden prediction. Chapter 5's gap decomposition (Eq. 5.1) is an inequality with a residual 'miscellaneous' term; while such a residual can in principle absorb slack, the paper provides independent theoretical results (e.g., monotone calibration cannot re-rank) and controlled empirical isolation of components. The integral of the gap is explicitly acknowledged to be equivalent to the existing E-AURC metric (Geifman et al., 2018), so there is no renaming of a known result as novel. Overall, the central claim that training dynamics provide a cheap, post-hoc abstention signal is supported by external comparisons and is not equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

Core assumptions: SPTD relies on a heuristic coupling between SGD dynamics and posterior uncertainty; the DP analysis relies on standard DP composition; the gap decomposition is a claimed theorem whose proof was not fully inspected here.

free parameters (5)
  • k (checkpoint weighting exponent) = k in [1,3] selected empirically
    Controls how strongly late-training disagreements are penalized in SPTD; Figure 3.6 shows k in [1,3] performs best, suggesting tuning on the benchmark.
  • number of checkpoints T = typically 25-50 (subsampling to 10 works at high coverage)
    Checkpointing resolution is a design choice; full trajectory gives best low-coverage performance.
  • selection threshold tau = varies by coverage level, set on validation
    Standard threshold for accept/reject in selective prediction; not a scientific constant but a fitted operating point.
  • Mirage loss tradeoff lambda = lambda tradeoff introduced in Appendix D.3.1
    Balances attack utility against inducing uncertainty; likely tuned per dataset.
  • DP-SGD noise multiplier and clipping norm = noise multiplier chosen adaptively to meet epsilon; clipping norm c=10
    Privacy mechanism parameters, not fitted to improve selective prediction, but they affect uncertainty quality.
assumptions (5)
  • standard math Chebyshev inequality bounds the probability that a datapoint was in the training set (Lemma 3.3.1)
    Used to motivate the forging-based derivation of SPTD scores.
  • domain assumption SGD iterates sample from a tempered Bayesian posterior (Mandt et al., 2017; Zhang et al., 2019)
    Section 3.4.1 and Eq. 3.12: checkpoint disagreement is interpreted as posterior predictive variance; this is the theoretical bridge from optimization noise to uncertainty.
  • ad hoc to paper The expected disagreement e_t is approximately 0 throughout training
    Section 3.3.3: 'by assuming e_t=0 always' simplifies the Chebyshev bound into the max and sum scores; the text says removing the assumption gives similar performance.
  • standard math Differential privacy post-processing and composition properties (Dwork et al., 2006)
    Section 4.3.2 classifies SC methods by whether they are post-processing, direct optimization, or sequential composition; this is standard DP theory.
  • domain assumption Five-way additive decomposition of the selective gap is valid with non-overlapping terms (Eq. 5.1)
    The finite-sample decomposition is the chapter's theorem; its validity as an inequality with interpretable terms is assumed/incomplete in the truncated text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning." pith.science (2026). https://pith.science/paper/TBCUVOS4

@misc{pith2026250807556,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBCUVOS4}},
  note         = {Machine review of arXiv:2508.07556}
}
read the original abstract

Machine learning (ML) systems are increasingly deployed in high-stakes domains where reliability is paramount. This thesis investigates how uncertainty estimation can enhance the safety and trustworthiness of ML, focusing on selective prediction -- where models abstain when confidence is low. We first show that a model's training trajectory contains rich uncertainty signals that can be exploited without altering its architecture or loss. By ensembling predictions from intermediate checkpoints, we propose a lightweight, post-hoc abstention method that works across tasks, avoids the cost of deep ensembles, and achieves state-of-the-art selective prediction performance. Crucially, this approach is fully compatible with differential privacy (DP), allowing us to study how privacy noise affects uncertainty quality. We find that while many methods degrade under DP, our trajectory-based approach remains robust, and we introduce a framework for isolating the privacy-uncertainty trade-off. Next, we then develop a finite-sample decomposition of the selective classification gap -- the deviation from the oracle accuracy-coverage curve -- identifying five interpretable error sources and clarifying which interventions can close the gap. This explains why calibration alone cannot fix ranking errors, motivating methods that improve uncertainty ordering. Finally, we show that uncertainty signals can be adversarially manipulated to hide errors or deny service while maintaining high accuracy, and we design defenses combining calibration audits with verifiable inference. Together, these contributions advance reliable ML by improving, evaluating, and safeguarding uncertainty estimation, enabling models that not only make accurate predictions -- but also know when to say "I do not know".

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

181 extracted references · 40 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    B., Mironov, I., Talwar, K., and Zhang, L

    Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.\ 308--318, 2016

    2016

  3. [3]

    and Kang, D

    Adila, D. and Kang, D. Understanding out-of-distribution: A perspective of data dynamics. In I (Still) Can't Believe It's Not Better! Workshop at NeurIPS 2021, pp.\ 1--8. PMLR, 2022

    2021

  4. [4]

    Estimating example difficulty using variance of gradients

    Agarwal, C., D'souza, D., and Hooker, S. Estimating example difficulty using variance of gradients. arXiv preprint arXiv:2008.11600, 2020

    arXiv 2008

  5. [5]

    C., Rangapuram, S., Salinas, D., Schulz, J., et al

    Alexandrov, A., Benidis, K., Bohlke-Schneider, M., Flunkert, V., Gasthaus, J., Januschowski, T., Maddix, D. C., Rangapuram, S., Salinas, D., Schulz, J., et al. Gluonts: Probabilistic time series models in python. arXiv preprint arXiv:1906.05264, 2019

    arXiv 1906

  6. [6]

    Concrete problems in ai safety

    Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  7. [7]

    Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021

    arXiv 2021

  8. [8]

    Baek, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35: 0 19274--19289, 2022

    2022

Show all 181 references
  1. [9]

    Differential privacy has disparate impact on model accuracy

    Bagdasaryan, E., Poursaeed, O., and Shmatikov, V. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32, 2019

    2019

  2. [10]

    Deep learning through the lens of example difficulty

    Baldock, R., Maennel, H., and Neyshabur, B. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34, 2021

    2021

  3. [11]

    QSAR fish toxicity

    Ballabio, D., Cassotti, M., Consonni, V., and Todeschini, R. QSAR fish toxicity . UCI Machine Learning Repository, 2019. DOI : https://doi.org/10.24432/C5JG7B

    2019 doi

  4. [12]

    Window-based distribution shift detection for deep neural networks

    Bar-Shalom, G., Geifman, Y., and El-Yaniv, R. Window-based distribution shift detection for deep neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  5. [13]

    Bartlett, P. L. and Wegkamp, M. H. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9 0 (8), 2008

    2008

  6. [14]

    Private empirical risk minimization: Efficient algorithms and tight error bounds

    Bassily, R., Smith, A., and Thakurta, A. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pp.\ 464--473. IEEE, 2014

    2014

  7. [15]

    Stability of stochastic gradient descent on nonsmooth convex losses

    Bassily, R., Feldman, V., Guzm \'a n, C., and Talwar, K. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33: 0 4381--4391, 2020

    2020

  8. [16]

    and Kohavi, R

    Becker, B. and Kohavi, R. Adult . UCI Machine Learning Repository, 1996. DOI : https://doi.org/10.24432/C5XW20

    1996 doi

  9. [17]

    Berger, J. O. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013

    2013

  10. [18]

    Bishop, C. M. Pattern recognition and machine learning. Springer, 2: 0 1122--1128, 2006

    2006

  11. [19]

    T., and Nakkiran, P

    B asiok, J., Gopalan, P., Hu, L., Kalai, A. T., and Nakkiran, P. Loss Minimization Yields Multicalibration for Large Neural Networks . In Guruswami, V. (ed.), 15th Innovations in Theoretical Computer Science Conference (ITCS 2024), volume 287 of Leibniz International Proceedin...

    2024 doi

  12. [20]

    M., Kucukelbir, A., and McAuliffe, J

    Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: A review for statisticians. Journal of the American statistical Association, 112 0 (518): 0 859--877, 2017

    2017

  13. [21]

    Weight uncertainty in neural network

    Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference on machine learning, pp.\ 1613--1622. PMLR, 2015

    2015

  14. [22]

    Food-101 -- mining discriminative components with random forests

    Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 -- mining discriminative components with random forests. In European Conference on Computer Vision, 2014

    2014

  15. [23]

    and Honaker, J

    Brawner, T. and Honaker, J. Bootstrap inference and differential privacy: Standard errors for free. Unpublished Manuscript, 2018

    2018

  16. [24]

    Bagging predictors

    Breiman, L. Bagging predictors. Machine learning, 24: 0 123--140, 1996

    1996

  17. [25]

    Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78 0 (1): 0 1--3, 1950

    1950

  18. [26]

    Universally composable security: a new paradigm for cryptographic protocols

    Canetti, R. Universally composable security: a new paradigm for cryptographic protocols. In Proceedings 42nd IEEE Symposium on Foundations of Computer Science, pp.\ 136--145, 2001. doi:10.1109/SFCS.2001.959888

    2001

  19. [27]

    Cattelan, L. F. P. and Silva, D. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. 2023 a

    2023

  20. [28]

    Cattelan, L. F. P. and Silva, D. Improving selective classification performance of deep neural networks through post-hoc logit normalization and temperature scaling. arXiv preprint arXiv:2305.15508, 2023 b

    2023 arXiv

  21. [29]

    Artificial intelligence, bias and clinical safety

    Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., and Tsaneva-Atanasova, K. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28 0 (3): 0 231--237, 2019

    2019

  22. [30]

    Active bias: Training more accurate neural networks by emphasizing high variance samples

    Chang, H.-S., Learned-Miller, E., and McCallum, A. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances in Neural Information Processing Systems, 30, 2017

    2017

  23. [31]

    Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019

    2019

  24. [32]

    Checkpoint ensembles: Ensemble methods from a single training process

    Chen, H., Lundberg, S., and Lee, S.-I. Checkpoint ensembles: Ensemble methods from a single training process. arXiv preprint arXiv:1710.03282, 2017

    2017 arXiv

  25. [33]

    An optimum character recognition system using decision functions

    Chow, C.-K. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, 0 (4): 0 247--254, 1957

    1957

  26. [34]

    Coenen, L., Abdullah, A. K. A., and Guns, T. Probability of default estimation, with a reject option. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp.\ 439--448, 2020. doi:10.1109/DSAA49011.2020.00058

    2020

  27. [35]

    Learning with rejection

    Cortes, C., DeSalvo, G., and Mohri, M. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp.\ 67--82. Springer, 2016

    2016

  28. [36]

    Unbiased statistical estimation and valid confidence intervals under differential privacy

    Covington, C., He, X., Honaker, J., and Kamath, G. Unbiased statistical estimation and valid confidence intervals under differential privacy. arXiv preprint arXiv:2110.14465, 2021

    2021 arXiv

  29. [37]

    Multiparty computation from somewhat homomorphic encryption

    Damg rd, I., Pastro, V., Smart, N., and Zakarias, S. Multiparty computation from somewhat homomorphic encryption. In Safavi-Naini, R. and Canetti, R. (eds.), Advances in Cryptology -- CRYPTO 2012, pp.\ 643--662, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-...

    2012

  30. [38]

    A probabilistic theory of pattern recognition, volume 31

    Devroye, L., Gy \"o rfi, L., and Lugosi, G. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013

    2013

  31. [39]

    Retiring adult: New datasets for fair machine learning

    Ding, F., Hardt, M., Miller, J., and Schmidt, L. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34: 0 6478--6490, 2021

    2021

  32. [40]

    Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off

    Ding, Y., Liu, J., Xiong, J., and Shi, Y. Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.\ 4--5, 2020

    2020

  33. [41]

    Calibrating noise to sensitivity in private data analysis

    Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.\ 265--284. Springer, 2006

    2006

  34. [42]

    The algorithmic foundations of differential privacy

    Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9 0 (3--4): 0 211--407, 2014

    2014

  35. [43]

    A., and Papernot, N

    Dziedzic, A., Rabanser, S., Yaghini, M., Ale, A., Erdogdu, M. A., and Papernot, N. p -DkNN: Out-of-Distribution Detection Through Statistical Testing of Deep Representations . arXiv preprint arXiv:2207.12545, 2022

    2022 arXiv

  36. [44]

    and Wiener, Y

    El-Yaniv, R. and Wiener, Y. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 0 (53): 0 1605--1641, 2010. URL http://jmlr.org/papers/v11/el-yaniv10a.html

    2010

  37. [45]

    El-Yaniv, R. et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 0 (5), 2010

    2010

  38. [46]

    Statistically valid inferences from privacy protected data

    Evans, G., King, G., Schwenzfeier, M., and Thakurta, A. Statistically valid inferences from privacy protected data. American Political Science Review, 2019

    2019

  39. [47]

    and Zhang, C

    Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 0 2881--2891, 2020

    2020

  40. [48]

    Selective prediction-set models with coverage rate guarantees

    Feng, J., Sondhi, A., Perry, J., and Simon, N. Selective prediction-set models with coverage rate guarantees. Biometrics, 2021

    2021

  41. [49]

    O., Hajimirsadeghi, H., and Abdi, A

    Feng, L., Ahmed, M. O., Hajimirsadeghi, H., and Abdi, A. H. Towards better selective classification. In The Eleventh International Conference on Learning Representations, 2023

    2023

  42. [50]

    Parametric bootstrap for differentially private confidence intervals

    Ferrando, C., Wang, S., and Sheldon, D. Parametric bootstrap for differentially private confidence intervals. In International Conference on Artificial Intelligence and Statistics, pp.\ 1598--1618. PMLR, 2022

    2022

  43. [51]

    Conformal prediction: a unified review of theory and new challenges

    Fontana, M., Zeni, G., and Vantini, S. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29 0 (1): 0 1--23, 2023

    2023

  44. [52]

    A., Thomas, M

    Franzese, N., Dziedzic, A., Choquette-Choo, C. A., Thomas, M. R., Kaleem, M. A., Rabanser, S., Fang, C., Jha, S., Papernot, N., and Wang, X. Robust and Actively Secure Serverless Collaborative Learning . Advances in Neural Information Processing Systems, 36: 0 39504--39528, 2023

    2023

  45. [53]

    Constant-overhead zero-knowledge for RAM programs

    Franzese, O., Katz, J., Lu, S., Ostrovsky, R., Wang, X., and Weng, C. Constant-overhead zero-knowledge for RAM programs. Cryptology ePrint Archive, Paper 2021/979, 2021. URL https://eprint.iacr.org/2021/979

    2021

  46. [54]

    and Ghahramani, Z

    Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp.\ 1050--1059. PMLR, 2016

    2016

  47. [55]

    Gal, Y. et al. Uncertainty in deep learning. 2016

    2016

  48. [56]

    What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers

    Galil, I., Dabbah, M., and El-Yaniv, R. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers. arXiv preprint arXiv:2302.11874, 2023

    2023 arXiv

  49. [57]

    Selective classification via one-sided prediction

    Gangrade, A., Kag, A., and Saligrama, V. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics, pp.\ 2179--2187. PMLR, 2021

    2021

  50. [58]

    Experimenting with zero-knowledge proofs of training

    Garg, S., Goel, A., Jha, S., Mahloujifar, S., Mahmoody, M., Policharla, G.-V., and Wang, M. Experimenting with zero-knowledge proofs of training. Cryptology ePrint Archive, Paper 2023/1345, 2023. URL https://eprint.iacr.org/2023/1345

    2023

  51. [59]

    Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56 0 (Suppl 1): 0 1513--1589, 2023

    2023

  52. [60]

    and El-Yaniv, R

    Geifman, Y. and El-Yaniv, R. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017

    2017

  53. [61]

    and El-Yaniv, R

    Geifman, Y. and El-Yaniv, R. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning, pp.\ 2151--2159. PMLR, 2019

    2019

  54. [62]

    Bias-reduced uncertainty estimation for deep neural classifiers

    Geifman, Y., Uziel, G., and El-Yaniv, R. Bias-reduced uncertainty estimation for deep neural classifiers. arXiv preprint arXiv:1805.08206, 2018

    2018 arXiv

  55. [63]

    Geyer, C. J. Practical markov chain monte carlo. Statistical science, pp.\ 473--483, 1992

    1992

  56. [64]

    Ghodsi, Z., Hari, S. K. S., Frosio, I., Tsai, T., Troccoli, A., Keckler, S. W., Garg, S., and Anandkumar, A. Generating and characterizing scenarios for safety testing of autonomous vehicles. arXiv preprint arXiv:2103.07403, 2021

    2021 arXiv

  57. [65]

    and Raftery, A

    Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102 0 (477): 0 359--378, 2007

    2007

  58. [66]

    The knowledge complexity of interactive proof-systems

    Goldwasser, S., Micali, S., and Rackoff, C. The knowledge complexity of interactive proof-systems. In Proceedings of the Seventeenth Annual ACM Symposium on Theory of Computing, STOC '85, pp.\ 291–304, New York, NY, USA, 1985. Association for Computing Machinery. ISBN 08979115...

    1985

  59. [67]

    When does a predictor know its own loss? arXiv preprint arXiv:2502.20375, 2025

    Gollakota, A., Gopalan, P., Karan, A., Peale, C., and Wieder, U. When does a predictor know its own loss? arXiv preprint arXiv:2502.20375, 2025

    2025 arXiv

  60. [68]

    Bounded-abstaining classification for breast tumors in imbalanced ultrasound images

    Guan, H., Zhang, Y., Cheng, H.-D., and Tang, X. Bounded-abstaining classification for breast tumors in imbalanced ultrasound images. International Journal of Applied Mathematics and Computer Science, 30 0 (2), 2020

    2020

  61. [69]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International conference on machine learning, pp.\ 1321--1330. PMLR, 2017

    2017

  62. [70]

    Scalable zero-knowledge proofs for non-linear functions in machine learning

    Hao, M., Chen, H., Li, H., Weng, C., Zhang, Y., Yang, H., and Zhang, T. Scalable zero-knowledge proofs for non-linear functions in machine learning. In 33rd USENIX Security Symposium (USENIX Security 24), pp.\ 3819--3836, Philadelphia, PA, August 2024. USENIX Association. ISBN...

    2024

  63. [71]

    Equality of opportunity in supervised learning

    Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016 a

    2016

  64. [72]

    Train faster, generalize better: Stability of stochastic gradient descent

    Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp.\ 1225--1234. PMLR, 2016 b

    2016

  65. [73]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  66. [74]

    Multicalibration: Calibration for the (computationally-identifiable) masses

    H \'e bert-Johnson, U., Kim, M., Reingold, O., and Rothblum, G. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning, pp.\ 1939--1948. PMLR, 2018

    1939

  67. [75]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019

    2019

  68. [76]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016

    2016 arXiv

  69. [77]

    Scaling out-of-distribution detection for real-world settings

    Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019

    1911 arXiv

  70. [78]

    Statlog (German Credit Data)

    Hofmann, H. Statlog (German Credit Data) . UCI Machine Learning Repository, 1994. DOI : https://doi.org/10.24432/C5NC77

    1994 doi

  71. [79]

    What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05248, 2019

    Hooker, S., Courville, A., Clark, G., Dauphin, Y., and Frome, A. What do compressed deep neural networks forget? arXiv preprint arXiv:1911.05248, 2019

    1911 arXiv

  72. [80]

    Detection of traffic signs in real-world images: The G erman T raffic S ign D etection B enchmark

    Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., and Igel, C. Detection of traffic signs in real-world images: The G erman T raffic S ign D etection B enchmark. In International Joint Conference on Neural Networks, number 1288, 2013

    2013

  73. [81]

    E., and Weinberger, K

    Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q. Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv:1704.00109, 2017

    2017 arXiv

  74. [82]

    Self-adaptive training: beyond empirical risk minimization

    Huang, L., Zhang, C., and Zhang, H. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33: 0 19365--19376, 2020

    2020

  75. [83]

    and Waegeman, W

    H \"u llermeier, E. and Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110 0 (3): 0 457--506, 2021

    2021

  76. [84]

    forecasting with exponential smoothing

    Hyndman, R. expsmooth: Data sets from “forecasting with exponential smoothing”. R package version, 2, 2015

    2015

  77. [85]

    Jiang, Y., Nagarajan, V., Baek, C., and Kolter, J. Z. Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106.13799, 2021

    2021 arXiv

  78. [86]

    D., and Yuan, B

    Jiang, Y., Fu, F., Zhao, W., Rabanser, S., Lane, N. D., and Yuan, B. Cascadia: A Cascade Serving System for Large Language Models . arXiv preprint arXiv:2506.04203, 2025

    2025

  79. [87]

    Jiang, Z., Zhang, C., Talwar, K., and Mozer, M. C. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206, 2020

    2002 arXiv

  80. [88]

    W., Kumar, A., and Liang, P

    Jones, E., Sagawa, S., Koh, P. W., Kumar, A., and Liang, P. Selective classification can magnify disparities across groups. arXiv preprint arXiv:2010.14134, 2020

    2010 arXiv

  81. [89]

    and Vadhan, S

    Karwa, V. and Vadhan, S. Finite sample differentially private confidence intervals. arXiv preprint arXiv:1711.03908, 2017

    2017 arXiv

  82. [90]

    Unanimous prediction for 100\ semantic mappings

    Khani, F., Rinard, M., and Liang, P. Unanimous prediction for 100\ semantic mappings. arXiv preprint arXiv:1606.06368, 2016

    2016 arXiv

  83. [91]

    Position: Uncertainty quantification needs reassessment for large-language model agents

    Kirchhof, M., Kasneci, G., and Kasneci, E. Position: Uncertainty quantification needs reassessment for large-language model agents. arXiv preprint arXiv:2505.22655, 2025

    2025 arXiv

  84. [92]

    W., Sagawa, S., Marklund, H., Xie, S

    Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp.\ 5637--5664. PMLR, 2021

    2021

  85. [93]

    Kompa, B., Snoek, J., and Beam, A. L. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 4 0 (1): 0 4, 2021

    2021

  86. [94]

    Empirical data drift detection experiments on real-world medical imaging data

    Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., and Abdalla, M. Empirical data drift detection experiments on real-world medical imaging data. Nature Communications, 15 0 (1): 0 1887, 2024

    2024

  87. [95]

    and Arce, G

    Kotropoulos, C. and Arce, G. R. Linear classifier with reject option for the detection of vocal fold paralysis and vocal fold edema. EURASIP Journal on Advances in Signal Processing, 2009: 0 1--13, 2009

    2009

  88. [96]

    3d object representations for fine-grained categorization

    Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp.\ 554--561, 2013

    2013

  89. [97]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  90. [98]

    Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration

    Kull, M., Perello Nieto, M., K \"a ngsepp, M., Silva Filho, T., Song, H., and Flach, P. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019

    2019

  91. [99]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processin...

    2017

  92. [100]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017 b

    2017

  93. [101]

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks

    Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018

    2018

  94. [102]

    vcnn: Verifiable convolutional neural network based on zk-snarks

    Lee, S., Ko, H., Kim, J., and Oh, H. vcnn: Verifiable convolutional neural network based on zk-snarks. IEEE Trans. Dependable Secur. Comput., 21 0 (4): 0 4254–4270, January 2024. ISSN 1545-5971. doi:10.1109/TDSC.2023.3348760. URL https://doi.org/10.1109/TDSC.2023.3348760

    2024

  95. [103]

    Trustworthy ai: From principles to practices

    Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., and Zhou, B. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55 0 (9): 0 1--46, 2023

    2023

  96. [104]

    Simple and principled uncertainty estimation with deterministic deep learning via distance awareness

    Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T., and Lakshminarayanan, B. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33: 0 7498--7512, 2020

    2020

  97. [105]

    Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis

    Liu, J., Gallego, B., and Barbieri, S. Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Scientific reports, 12 0 (1): 0 1762, 2022

    2022

  98. [106]

    P., Salakhutdinov, R

    Liu, Z., Wang, Z., Liang, P. P., Salakhutdinov, R. R., Morency, L.-P., and Ueda, M. Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32, 2019

    2019

  99. [107]

    Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward

    Lo Piano, S. Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward. Humanities and Social Sciences Communications, 7 0 (1): 0 1--7, 2020

    2020

  100. [108]

    Certifiers make neural networks vulnerable to availability attacks

    Lorenz, T., Kwiatkowska, M., and Fritz, M. Certifiers make neural networks vulnerable to availability attacks. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pp.\ 67--78, 2023

    2023

  101. [109]

    The m4 competition: 100,000 time series and 61 forecasting methods

    Makridakis, S., Spiliotis, E., and Assimakopoulos, V. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36 0 (1): 0 54--74, 2020

    2020

  102. [110]

    D., and Blei, D

    Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18 0 (134): 0 1--35, 2017

    2017

  103. [111]

    A survey on bias and fairness in machine learning

    Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54 0 (6): 0 1--35, 2021

    2021

  104. [112]

    R \'e nyi differential privacy

    Mironov, I. R \'e nyi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp.\ 263--275. IEEE, 2017

    2017

  105. [113]

    and Sontag, D

    Mozannar, H. and Sontag, D. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pp.\ 7076--7087. PMLR, 2020

    2020

  106. [114]

    Integral probability metrics and their generating classes of functions

    M \"u ller, A. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29 0 (2): 0 429--443, 1997

    1997

  107. [115]

    P., Cooper, G., and Hauskrecht, M

    Naeini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015

    2015

  108. [116]

    Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. 2011

    2011

  109. [117]

    and Caruana, R

    Niculescu-Mizil, A. and Caruana, R. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp.\ 625--632, 2005

    2005

  110. [118]

    B., Nordholt, P

    Nielsen, J. B., Nordholt, P. S., Orlandi, C., and Burra, S. S. A new approach to practical active-secure two-party computation. In Safavi-Naini, R. and Canetti, R. (eds.), Advances in Cryptology -- CRYPTO 2012, pp.\ 681--700, Berlin, Heidelberg, 2012. Springer Berlin Heidelber...

    2012

  111. [119]

    Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift

    Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019

    2019

  112. [120]

    Pace, R. K. and Barry, R. Sparse spatial autoregressions. Statistics & Probability Letters, 33 0 (3): 0 291--297, 1997

    1997

  113. [121]

    B., and Swami, A

    Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., and Swami, A. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp.\ 506--519, 2017

    2017

  114. [122]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An Impera...

    2019

  115. [123]

    Came: Contrastive automated model evaluation

    Peng, R., Duan, Q., Wang, H., Ma, J., Jiang, Y., Tu, Y., Jiang, X., and Zhao, J. Came: Contrastive automated model evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 20121--20132, 2023

    2023

  116. [124]

    Energy-based automated model evaluation

    Peng, R., Zou, H., Wang, H., Zeng, Y., Huang, Z., and Zhao, J. Energy-based automated model evaluation. arXiv preprint arXiv:2401.12689, 2024

    2024 arXiv

  117. [125]

    Examining the challenges in development data pipeline

    Pervaiz, F., Vashistha, A., and Anderson, R. Examining the challenges in development data pipeline. In Proceedings of the 2nd ACM SIGCAS Conference on Computing and Sustainable Societies, pp.\ 13--21, 2019

    2019

  118. [126]

    Platt, J. et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10 0 (3): 0 61--74, 1999

    1999

  119. [127]

    Potra, F. A. and Wright, S. J. Interior-point methods. Journal of computational and applied mathematics, 124 0 (1-2): 0 281--302, 2000

    2000

  120. [128]

    Suitability Filter: A Statistical Framework for Model Evaluation in Real-World Deployment Settings

    Pouget, A., Yaghini, M., Rabanser, S., and Papernot, N. Suitability Filter: A Statistical Framework for Model Evaluation in Real-World Deployment Settings . In Proceedings of the International Conference on Machine Learning (ICML) , 2025

    2025

  121. [129]

    and Ruggieri, S

    Pugnana, A. and Ruggieri, S. Auc-based selective classification. In International Conference on Artificial Intelligence and Statistics, pp.\ 2494--2514. PMLR, 2023

    2023

  122. [130]

    and Papernot, N

    Rabanser, S. and Papernot, N. What Does It Take to Build a Performant Selective Classifier? In Submission, 2025

    2025

  123. [131]

    Intrinsic Anomaly Detection for Multi-Variate Time Series

    Rabanser, S., Januschowski, T., Rasul, K., Borchert, O., Kurle, R., Gasthaus, J., Bohlke-Schneider, M., Papernot, N., and Flunkert, V. Intrinsic Anomaly Detection for Multi-Variate Time Series . arXiv preprint arXiv:2206.14342, 2022

    2022 arXiv

  124. [132]

    Training Private Models That Know What They Don't Know

    Rabanser, S., Thudi, A., Guha Thakurta, A., Dvijotham, K., and Papernot, N. Training Private Models That Know What They Don't Know . Advances in Neural Information Processing Systems, 36: 0 53711--53727, 2023

    2023

  125. [133]

    Gatekeeper: Improving Model Cascades Through Confidence Tuning

    Rabanser, S., Rauschmayr, N., Kulshrestha, A., Poklukar, P., Jitkrittum, W., Augenstein, S., Wang, C., and Tombari, F. Gatekeeper: Improving Model Cascades Through Confidence Tuning . In Tiny Titans: The next wave of On-Device Learning for Foundational Models (TTODLer-FM) Work...

    2025

  126. [134]

    S., Franzese, O., Wang, X., Weller, A., and Papernot, N

    Rabanser, S., Shamsabadi, A. S., Franzese, O., Wang, X., Weller, A., and Papernot, N. Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention . In Proceedings of the 42nd International Conference on Machine Learning, 2025 b

    2025

  127. [135]

    Selective Prediction Via Training Dynamics

    Rabanser, S., Thudi, A., Hamidieh, K., Dziedzic, A., and Papernot, N. Selective Prediction Via Training Dynamics . Transactions on Machine Learning Research, 2025 c

    2025

  128. [136]

    A general framework for detecting anomalous inputs to dnn classifiers

    Raghuram, J., Chandrasekaran, V., Jha, S., and Banerjee, S. A general framework for detecting anomalous inputs to dnn classifiers. In International Conference on Machine Learning, pp.\ 8764--8775. PMLR, 2021

    2021

  129. [137]

    and Mehrotra, S

    Rahimian, H. and Mehrotra, S. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization, 3: 0 1--85, 2022

    2022

  130. [138]

    G., Padhy, S., and Lakshminarayanan, B

    Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lakshminarayanan, B. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022, 2021

    2021 arXiv

  131. [139]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

    Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1 0 (5): 0 206--215, 2019

    2019

  132. [140]

    A., Binder, A., M \"u ller, E., and Kloft, M

    Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S. A., Binder, A., M \"u ller, E., and Kloft, M. Deep one-class classification. In International conference on machine learning, pp.\ 4393--4402. PMLR, 2018

    2018

  133. [141]

    Deepar: Probabilistic forecasting with autoregressive recurrent networks

    Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36 0 (3): 0 1181--1191, 2020

    2020

  134. [142]

    Expert load matters: operating networks at high accuracy and low manual effort

    Sangalli, S., Erdil, E., and Konukoglu, E. Expert load matters: operating networks at high accuracy and low manual effort. Advances in Neural Information Processing Systems, 36, 2024

    2024

  135. [143]

    Schein, A. I. and Ungar, L. H. Active learning for logistic regression: an evaluation. Machine Learning, 68 0 (3): 0 235--265, 2007

    2007

  136. [144]

    and Vovk, V

    Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9 0 (3), 2008

    2008

  137. [145]

    and Ben-David, S

    Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  138. [146]

    Recycling scraps: Improving private learning by leveraging intermediate checkpoints

    Shejwalkar, V., Ganesh, A., Mathews, R., Thakkar, O., and Thakurta, A. Recycling scraps: Improving private learning by leveraging intermediate checkpoints. arXiv preprint arXiv:2210.01864, 2022

    2022 arXiv

  139. [147]

    Introduction to the theory of computation

    Sipser, M. Introduction to the theory of computation. ACM Sigact News, 27 0 (1): 0 27--29, 1996

    1996

  140. [148]

    Sousa, R., Mora, B., and Cardoso, J. S. An ordinal data method for the classification with reject option. In 2009 International Conference on Machine Learning and Applications, pp.\ 746--750. IEEE, 2009

    2009

  141. [149]

    Dropout: a simple way to prevent neural networks from overfitting

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 0 (1): 0 1929--1958, 2014

    1929

  142. [150]

    Steinhardt, J., Koh, P. W. W., and Liang, P. S. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017

    2017

  143. [151]

    zkllm: Zero knowledge proofs for large language models

    Sun, H., Li, J., and Zhang, H. zkllm: Zero knowledge proofs for large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS '24, pp.\ 4405–4419, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 97984...

    2024

  144. [152]

    Out-of-distribution detection with deep nearest neighbors

    Sun, Y., Ming, Y., Zhu, X., and Li, Y. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pp.\ 20827--20840. PMLR, 2022

    2022

  145. [153]

    A., and Choi, Y

    Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., and Choi, Y. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795, 2020

    2009 arXiv

  146. [154]

    Intriguing properties of neural networks

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013

    2013 arXiv

  147. [155]

    Rethinking the inception architecture for computer vision

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2818--2826, 2016

    2016

  148. [156]

    On the necessity of auditable algorithmic definitions for machine unlearning

    Thudi, A., Jia, H., Shumailov, I., and Papernot, N. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pp.\ 4007--4022, 2022

    2022

  149. [157]

    Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018

    2018 arXiv

  150. [158]

    I., Papadimitriou, E., and van Gelder, P

    Tselentis, D. I., Papadimitriou, E., and van Gelder, P. The usefulness of artificial intelligence for safety assessment of different transport modes. Accident Analysis & Prevention, 186: 0 107034, 2023

    2023

  151. [159]

    N., O’Hagan, M., and O’Sullivan, C

    Vieira, L. N., O’Hagan, M., and O’Sullivan, C. Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases. Information, Communication & Society, 24 0 (11): 0 1515--1532, 2021

    2021

  152. [160]

    A., and Kumar, A

    Vijh, M., Chandola, D., Tikkiwal, V. A., and Kumar, A. Stock closing price prediction using machine learning techniques. Procedia computer science, 167: 0 599--606, 2020

    2020

  153. [161]

    Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., and Zhao, B. Y. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE symposium on security and privacy (SP), pp.\ 707--723. IEEE, 2019

    2019

  154. [162]

    In pursuit of interpretable, fair and accurate machine learning for criminal recidivism prediction

    Wang, C., Han, B., Patel, B., and Rudin, C. In pursuit of interpretable, fair and accurate machine learning for criminal recidivism prediction. Journal of Quantitative Criminology, 39 0 (2): 0 519--581, 2023

    2023

  155. [163]

    J., and Katz, J

    Wang, X., Malozemoff, A. J., and Katz, J. EMP-toolkit: Efficient MultiParty computation toolkit . https://github.com/emp-toolkit, 2016

    2016

  156. [164]

    Learning with noisy labels revisited: A study using real-world human annotations

    Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:2110.12088, 2021

    2021 arXiv

  157. [165]

    Wolverine: Fast, scalable, and communication-efficient zero-knowledge proofs for boolean and arithmetic circuits

    Weng, C., Yang, K., Katz, J., and Wang, X. Wolverine: Fast, scalable, and communication-efficient zero-knowledge proofs for boolean and arithmetic circuits. In 2021 IEEE Symposium on Security and Privacy (SP), pp.\ 1074--1091, 2021 a . doi:10.1109/SP40001.2021.00056

    2021

  158. [166]

    Mystique: Efficient conversions for \ Zero-Knowledge \ proofs with applications to machine learning

    Weng, C., Yang, K., Xie, X., Katz, J., and Wang, X. Mystique: Efficient conversions for \ Zero-Knowledge \ proofs with applications to machine learning. In 30th USENIX Security Symposium (USENIX Security 21), pp.\ 501--518, 2021 b

    2021

  159. [167]

    and El-Yaniv, R

    Wiener, Y. and El-Yaniv, R. Agnostic selective classification. Advances in neural information processing systems, 24, 2011

    2011

  160. [168]

    X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al

    Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. Do no harm: a roadmap for responsible machine learning for health care. Nature medicine, 25 0 (9): 0 1337--1340, 2019

    2019

  161. [169]

    and McSherry, F

    Williams, O. and McSherry, F. Probabilistic inference and differential privacy. Advances in neural information processing systems, 23, 2010

    2010

  162. [170]

    Confidence-aware contrastive learning for selective classification

    Wu, Y.-C., Lyu, S.-H., Shang, H., Wang, X., and Qian, C. Confidence-aware contrastive learning for selective classification. arXiv preprint arXiv:2406.04745, 2024

    2024 arXiv

  163. [171]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  164. [172]

    Regulation games for trustworthy machine learning

    Yaghini, M., Liu, P., Boenisch, F., and Papernot, N. Regulation games for trustworthy machine learning. arXiv preprint arXiv:2402.03540, 2024

    2024 arXiv

  165. [173]

    Concrete Compressive Strength

    Yeh, I.-C. Concrete Compressive Strength . UCI Machine Learning Repository, 2007. DOI : https://doi.org/10.24432/C5PK67

    2007 doi

  166. [174]

    Opacus: U ser-friendly differential privacy library in PyTorch

    Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., Cormode, G., and Mironov, I. Opacus: U ser-friendly differential privacy library in PyTorch . arXiv preprint arXiv:2109.12298, 2021

    2021 arXiv

  167. [175]

    and Elkan, C

    Zadrozny, B. and Elkan, C. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, 2001

    2001

  168. [176]

    and Elkan, C

    Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp.\ 694--699, 2002

    2002

  169. [177]

    and Komodakis, N

    Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  170. [178]

    Regression with reject option and application to knn

    Zaoui, A., Denis, C., and Hebiri, M. Regression with reject option and application to knn. Advances in Neural Information Processing Systems, 33: 0 20073--20082, 2020

    2020

  171. [179]

    Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G. Cyclical stochastic gradient mcmc for bayesian deep learning. arXiv preprint arXiv:1902.03932, 2019

    1902 arXiv

  172. [180]

    Age progression/regression by conditional adversarial autoencoder

    Zhang, Z., Song, Y., and Qi, H. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017

    2017

  173. [181]

    Rethinking confidence calibration for failure prediction

    Zhu, F., Cheng, Z., Zhang, X.-Y., and Liu, C.-L. Rethinking confidence calibration for failure prediction. In European conference on computer vision, pp.\ 518--536. Springer, 2022

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.