Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning
read the original abstract
This monograph deals with adaptive supervised classification, using tools borrowed from statistical mechanics and information theory, stemming from the PACBayesian approach pioneered by David McAllester and applied to a conception of statistical learning theory forged by Vladimir Vapnik. Using convex analysis on the set of posterior probability measures, we show how to get local measures of the complexity of the classification model involving the relative entropy of posterior distributions with respect to Gibbs posterior measures. We then discuss relative bounds, comparing the generalization error of two classification rules, showing how the margin assumption of Mammen and Tsybakov can be replaced with some empirical measure of the covariance structure of the classification model.We show how to associate to any posterior distribution an effective temperature relating it to the Gibbs prior distribution with the same level of expected error rate, and how to estimate this effective temperature from data, resulting in an estimator whose expected error rate converges according to the best possible power of the sample size adaptively under any margin and parametric complexity assumptions. We describe and study an alternative selection scheme based on relative bounds between estimators, and present a two step localization technique which can handle the selection of a parametric model from a family of those. We show how to extend systematically all the results obtained in the inductive setting to transductive learning, and use this to improve Vapnik's generalization bounds, extending them to the case when the sample is made of independent non-identically distributed pairs of patterns and labels. Finally we review briefly the construction of Support Vector Machines and show how to derive generalization bounds for them, measuring the complexity either through the number of support vectors or through the value of the transductive or inductive margin.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data
Asymmetric Langevin Unlearning uses public data to suppress unlearning noise costs by O(1/n_pub²), enabling practical mass unlearning with preserved utility under distribution mismatch.
-
Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities
A unified data-processing framework produces tighter change-of-measure inequalities that improve information-theoretic generalization bounds across learning theory and privacy.
-
On the Generalization Error of Differentially Private Algorithms via Typicality
Sharper information-theoretic generalization bounds for differentially private algorithms obtained via typicality arguments that improve prior mutual-information results and add new maximal-leakage bounds.
-
Anytime PAC-Bayes for Constrained Density-Ratio Networks under Covariate Shift
A constrained density-ratio network with augmented-Lagrangian enforcement and anytime PAC-Bayes delivers generalization certificates for importance-weighted learning under covariate shift.
-
Anytime PAC-Bayes for Constrained Density-Ratio Networks under Covariate Shift
A constrained density-ratio network approximates the Radon-Nikodym derivative and feeds an anytime PAC-Bayes certificate for learning under covariate shift, validated via synthetic patch tests and real-data deployment.
-
PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory
PAC-Bayes bounds for Gibbs posteriors are obtained via singular learning theory, producing explicit and tighter posterior-averaged risk bounds that adapt to data structure in overparameterized models.
-
Chaining Meets Chain Rule: Multilevel Entropic Regularization and Training of Neural Nets
Derives algorithm-dependent generalization bounds for neural nets using multilevel entropic regularization and proposes a Metropolis-simulated multi-scale Gibbs training procedure tested on a two-layer net for MNIST.
-
The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems
Converts impossibility theorems into architecture-dependent accuracy ceilings and design rules for transformers and other AI subfields, with the Deterministic Horizon measured at 19-31 across twelve models.
-
Is K-fold cross validation the best model selection method for Machine Learning?
K-fold CUBV combines cross-validation with PAC-Bayesian upper bounds on actual risk to provide a more robust criterion for validating ML accuracy and reducing false positives than standard CV.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.