arxiv: 2602.07999 · v3 · submitted 2026-02-08 · 💻 cs.IT · cs.LG· math.IT

Recognition: 2 theorem links

· Lean Theorem

Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities

Yanxiao Liu , Yijun Fan , Deniz G\"und\"uz

Authors on Pith no claims yet

Pith reviewed 2026-05-16 06:06 UTC · model grok-4.3

classification 💻 cs.IT cs.LGmath.IT

keywords change of measure inequalitiesdata processing inequalitygeneralization boundsdifferential privacyf-divergencesRenyi divergencealpha-mutual informationPAC-Bayesian bounds

0 comments

The pith

Novel change of measure inequalities derived from the data processing inequality provide tighter bounds for generalization and privacy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors develop a unified framework to derive change of measure inequalities by applying the data processing inequality to various information measures. This yields new, tighter versions of these inequalities for f-divergences, Renyi divergence, and alpha-mutual information. The results are then used to strengthen guarantees in generalization error analysis, PAC-Bayesian bounds, differential privacy, and analysis of data memorization. Sympathetic readers would care because these tighter bounds offer improved theoretical tools for understanding and designing learning algorithms with better performance guarantees.

Core claim

We propose novel change of measure inequalities via a unified framework based on the data processing inequality. This elementary yet powerful approach provides change of measure inequalities in terms of a broad family of information measures, including f-divergences, Renyi divergence, and alpha-mutual information. When applied to generalization error analysis, PAC-Bayesian theory, differential privacy, and data memorization, the new inequalities deliver stronger guarantees while recovering known results through simplified analyses.

What carries the argument

The unified framework based on the data processing inequality for deriving change of measure inequalities from divergences between probability measures.

If this is right

Tighter bounds on generalization error in machine learning models
Stronger PAC-Bayesian generalization guarantees
Improved differential privacy guarantees
Better analysis of data memorization in learning algorithms

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

These inequalities could potentially be extended to continuous or high-dimensional settings where traditional bounds are loose.
The simplified analyses might inspire similar elementary derivations in other areas of information-theoretic learning theory.
If adopted, practitioners could use these to derive tighter privacy budgets in real-world machine learning deployments.

Load-bearing premise

The data processing inequality can be applied directly to the family of information measures without additional regularity conditions that would undermine the tightness of the bounds in the target applications.

What would settle it

A specific example in a supervised learning task where the new bound on generalization error is violated or fails to be tighter than existing change of measure bounds.

Figures

Figures reproduced from arXiv: 2602.07999 by Deniz G\"und\"uz, Yanxiao Liu, Yijun Fan.

read the original abstract

Change of measure inequalities translate divergences between probability measures into explicit bounds on event probabilities, and play an important role in deriving probabilistic guarantees in learning theory, information theory, and statistics. We propose novel change of measure inequalities via a unified framework based on the data processing inequality, which is surprisingly elementary yet powerful enough to yield novel, tighter inequalities. We provide change of measure inequalities in terms of a broad family of information measures, including $f$-divergences (with Kullback-Leibler divergence and $\chi^2$-divergence as special cases), R\'enyi divergence, and $\alpha$-mutual information (with maximal leakage as a special case). We apply these results to generalization error analysis, PAC-Bayesian theory, differential privacy, and data memorization, obtaining stronger guarantees while recovering best-known results through simplified analyses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper gives a clean unified derivation of tighter change-of-measure inequalities from the data processing inequality.

read the letter

This paper gives a single elementary trick, based on the data processing inequality, that produces tighter change-of-measure bounds for f-divergences, Rényi divergence, and alpha-mutual information. The same framework recovers the best-known results as special cases while improving the constants in the generalization and privacy applications. What stands out is how straightforward the derivations are. Starting from DPI, they get explicit bounds without extra machinery, and then plug them into PAC-Bayesian analysis, differential privacy, and memorization bounds. The applications show stronger guarantees, and the analyses look simpler than the original proofs they replace. The math checks out on the surface: no circular definitions, no fitted parameters, and the tightness comes from the specific choice of measures rather than loose inequalities. The stress-test confirms that DPI applies directly here without hidden conditions that would weaken the claims. A minor soft spot is that the full numerical comparisons in the generalization sections rely on the detailed proofs, which aren't in the abstract. If those hold, the improvements are real; otherwise the gains might be smaller than advertised. But nothing suggests a load-bearing flaw. This work is for researchers who regularly use information-theoretic tools for generalization bounds or privacy analysis. It gives them a reusable technique that can tighten existing results without much extra effort. I would bring it to a reading group and cite it if the applications check out. It deserves peer review because the core contribution is clean and the applications are relevant to active areas.

Referee Report

1 major / 2 minor

Summary. The paper proposes a unified framework based on the data processing inequality to derive novel change of measure inequalities for a broad family of information measures, including f-divergences (with KL and χ² as special cases), Rényi divergence, and α-mutual information (with maximal leakage as a special case). These inequalities are applied to generalization error analysis, PAC-Bayesian theory, differential privacy, and data memorization to obtain stronger guarantees while recovering known results via simplified analyses.

Significance. If the derivations hold and the bounds are indeed tighter without additional regularity conditions, the work provides an elementary yet powerful tool for tightening information-theoretic guarantees in learning theory and privacy. The recovery of best-known results as special cases and the unified DPI-based approach strengthen its potential impact by simplifying existing analyses.

major comments (1)

The central claim of novel tighter inequalities rests on applying the data processing inequality directly to the chosen family of measures. The manuscript references proofs for these derivations and their tightness in the generalization and privacy applications, but these proofs are not included in the provided text, preventing full verification that no hidden regularity conditions affect the claimed improvements.

minor comments (2)

In the applications sections, explicit quantitative comparisons (e.g., numerical dominance or analytical gap to prior bounds) would better substantiate the 'tighter' claim beyond the abstract statement.
Notation for α-mutual information and its relation to maximal leakage should be defined more explicitly in the preliminaries to ensure consistency with standard definitions in the literature.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive assessment, the recommendation for minor revision, and the careful reading of the manuscript. The unified DPI-based framework indeed yields the claimed tighter inequalities without extra regularity conditions, as shown in the derivations. We address the major comment below.

read point-by-point responses

Referee: The central claim of novel tighter inequalities rests on applying the data processing inequality directly to the chosen family of measures. The manuscript references proofs for these derivations and their tightness in the generalization and privacy applications, but these proofs are not included in the provided text, preventing full verification that no hidden regularity conditions affect the claimed improvements.

Authors: We thank the referee for highlighting this point. The proofs appear in Appendix A, where each inequality is obtained by a direct application of the data-processing inequality to the chosen information measure (f-divergences, Rényi divergence, and α-mutual information) with no additional regularity assumptions beyond those already required for the measures to be well-defined. Tightness follows from explicit comparisons with existing bounds in Sections 4–7, where the new inequalities recover the best-known results as special cases and strictly improve them in general. To address the verification concern, we will insert a short outline of the core derivation steps into the main text (Section 3) and add a clarifying remark that no hidden conditions are used. This change improves accessibility without altering any stated results. revision: yes

Circularity Check

0 steps flagged

No significant circularity: derivation starts from standard DPI

full rationale

The paper constructs change-of-measure inequalities by applying the standard data processing inequality (DPI) to a family of information measures (f-divergences, Rényi divergence, α-mutual information). DPI is an established external result, not derived or redefined inside the paper. The resulting explicit bounds recover known results as special cases through direct substitution, without fitted parameters, self-referential definitions, or load-bearing self-citations that reduce the central claim to its own inputs. The framework is therefore self-contained against external benchmarks and introduces no circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the standard data-processing inequality for the chosen information measures and on the usual definitions of f-divergences, Rényi divergence, and α-mutual information; no free parameters or invented entities are introduced in the abstract.

axioms (1)

standard math Data processing inequality holds for the family of information measures considered
Invoked as the starting point for the unified framework in the abstract.

pith-pipeline@v0.9.0 · 5452 in / 1160 out tokens · 28094 ms · 2026-05-16T06:06:00.209834+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose novel change of measure inequalities via a unified framework based on the data processing inequality for f-divergences... Df(T∘P∥T∘Q)≤Df(P∥Q) with T=1_E yielding q f(p/q)+(1-q)f((1-p)/(1-q))
IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean costAlphaLog_high_calibrated_iff unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

f(t)=t log t recovers KL; f(t)=t²−1 recovers χ²; f(t)=[t−γ]+ recovers E_γ

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

90 extracted references · 90 canonical work pages · 4 internal anchors

[1]

Controlling bias in adaptive data analysis using information theory,

D. Russo and J. Zou, “Controlling bias in adaptive data analysis using information theory,” inArtificial Intelligence and Statistics. PMLR, 2016, pp. 1232–1240

work page 2016
[2]

Information-theoretic analysis of generalization capability of learning algorithms,

A. Xu and M. Raginsky, “Information-theoretic analysis of generalization capability of learning algorithms,”Advances in Neural Information Processing Systems, vol. 30, 2017

work page 2017
[3]

Generalization error bounds via Rényi-,f-divergences and maximal leakage,

A. R. Esposito, M. Gastpar, and I. Issa, “Generalization error bounds via Rényi-,f-divergences and maximal leakage,”IEEE Transactions on Information Theory, vol. 67, no. 8, pp. 4986–5004, 2021

work page 2021
[4]

Generalization bounds via information density and conditional information density,

F. Hellström and G. Durisi, “Generalization bounds via information density and conditional information density,”IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 3, pp. 824–839, 2020

work page 2020
[5]

An operational measure of information leakage,

I. Issa, S. Kamath, and A. B. Wagner, “An operational measure of information leakage,” inConference on Information Science and Systems (CISS). IEEE, 2016, pp. 234–239

work page 2016
[6]

f-divergence inequalities,

I. Sason and S. Verdú, “f-divergence inequalities,”IEEE Transactions on Information Theory, vol. 62, no. 11, pp. 5973–6006, 2016

work page 2016
[7]

Information radius,

R. Sibson, “Information radius,”Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, vol. 14, no. 2, pp. 149–160, 1969

work page 1969
[8]

α-mutual information,

S. Verdú, “α-mutual information,” inInformation Theory and Applications Workshop (ITA). IEEE, 2015, pp. 1–6

work page 2015
[9]

A primer on PAC-Bayesian learning,

B. Guedj, “A primer on PAC-Bayesian learning,”arXiv preprint arXiv:1901.05353, 2019

work page arXiv 1901
[10]

Reasoning about generalization via conditional mutual information,

T. Steinke and L. Zakynthinou, “Reasoning about generalization via conditional mutual information,” inConference on Learning Theory (COLT). PMLR, 2020, pp. 3437–3452

work page 2020
[11]

Calibrating noise to sensitivity in private data analysis,

C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” inThird Theory of Cryptography Conference, TCC 2006, New York, NY, March, 2006, pp. 265–284

work page 2006
[12]

Max-information, differential privacy, and post-selection hypothesis testing,

R. Rogers, A. Roth, A. Smith, and O. Thakkar, “Max-information, differential privacy, and post-selection hypothesis testing,” in2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 2016, pp. 487–494

work page 2016
[13]

I-divergence geometry of probability distributions and minimization problems,

I. Csiszár, “I-divergence geometry of probability distributions and minimization problems,”The Annals of Probability, pp. 146–158, 1975

work page 1975
[14]

Large deviations for markov processes and the asymptotic evaluation of certain markov process expectations for large times,

M. Donsker and S. Varadhan, “Large deviations for markov processes and the asymptotic evaluation of certain markov process expectations for large times,” inProbabilistic Methods in Differential Equations: Proceedings of the Conference Held at the University of Victoria, August 19–20, 1974. Springer, 2006, pp. 82–88

work page 1974
[15]

Change of measure through the Legendre transform

A. Picard-Weibel and B. Guedj, “On change of measure inequalities forf-divergences,”arXiv preprint arXiv:2202.05568, 2022

work page internal anchor Pith review Pith/arXiv arXiv 2022
[16]

Novel change of measure inequalities with applications to PAC-Bayesian bounds and monte carlo estimation,

Y . Ohnishi and J. Honorio, “Novel change of measure inequalities with applications to PAC-Bayesian bounds and monte carlo estimation,” inInternational conference on artificial intelligence and statistics. PMLR, 2021, pp. 1711–1719

work page 2021
[17]

Polyanskiy and Y

Y . Polyanskiy and Y . Wu,Information theory: From coding to learning. Cambridge University Press, 2025

work page 2025
[18]

Learners that use little information,

R. Bassily, S. Moran, I. Nachum, J. Shafer, and A. Yehudayoff, “Learners that use little information,” inAlgorithmic Learning Theory. PMLR, 2018, pp. 25–55

work page 2018
[19]

Chaining mutual information and tightening generalization bounds,

A. Asadi, E. Abbe, and S. Verdú, “Chaining mutual information and tightening generalization bounds,”Advances in Neural Information Processing Systems, vol. 31, 2018

work page 2018
[20]

f-divergences and their applications in lossy compression and bounding generalization error,

S. Masiha, A. Gohari, and M. H. Yassaee, “f-divergences and their applications in lossy compression and bounding generalization error,” IEEE Transactions on Information Theory, vol. 69, no. 12, pp. 7538–7564, 2023

work page 2023
[21]

A unified framework for information-theoretic generalization bounds,

Y . Chu and M. Raginsky, “A unified framework for information-theoretic generalization bounds,”Advances in Neural Information Processing Systems, vol. 36, pp. 79 260–79 278, 2023

work page 2023
[22]

Rate-distortion theoretic generalization bounds for stochastic learning algorithms,

M. Sefidgaran, A. Gohari, G. Richard, and U. Simsekli, “Rate-distortion theoretic generalization bounds for stochastic learning algorithms,” inConference on Learning Theory (COLT). PMLR, 2022, pp. 4416–4463

work page 2022
[23]

On measures of entropy and information,

A. Rényi, “On measures of entropy and information,” inProceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics. The Regents of the University of California, 1961

work page 1961
[24]

Generalization in adaptive data analysis and holdout reuse,

C. Dwork, V . Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth, “Generalization in adaptive data analysis and holdout reuse,” Advances in neural information processing systems, vol. 28, 2015

work page 2015
[25]

Preserving statistical validity in adaptive data analysis,

C. Dwork, V . Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. L. Roth, “Preserving statistical validity in adaptive data analysis,” in ACM Symposium on Theory of Computing, 2015, pp. 117–126

work page 2015
[26]

On information-type measure of difference of probability distributions and indirect observations,

I. Csiszár, “On information-type measure of difference of probability distributions and indirect observations,”Studia Sci. Math. Hungar., vol. 2, pp. 299–318, 1967. February 11, 2026 DRAFT 16

work page 1967
[27]

Eine informationstheoretische ungleichung und ihre anwendung auf den beweis der ergodizität von markoffschen ketten,

——, “Eine informationstheoretische ungleichung und ihre anwendung auf den beweis der ergodizität von markoffschen ketten,”A Magyar Tudományos Akadémia Matematikai Kutató Intézetének Közleményei, vol. 8, no. 1-2, pp. 85–108, 1963

work page 1963
[28]

A general class of coefficients of divergence of one distribution from another,

S. M. Ali and S. D. Silvey, “A general class of coefficients of divergence of one distribution from another,”Journal of the Royal Statistical Society: Series B (Methodological), vol. 28, no. 1, pp. 131–142, 1966

work page 1966
[29]

On the concept and measure of information contained in an observation,

I. Vincze, “On the concept and measure of information contained in an observation,” inContributions to probability. Elsevier, 1981, pp. 207–214

work page 1981
[30]

Le Cam,Asymptotic methods in statistical decision theory

L. Le Cam,Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012

work page 2012
[31]

A DPI-PAC-Bayesian framework for generalization bounds,

M. Guan, F. Farokhi, and J. Zhu, “A DPI-PAC-Bayesian framework for generalization bounds,” inIEEE Information Theory Workshop (ITW) 2025, Australia, 2025, pp. 1–6

work page 2025
[32]

Computable bounds on the exploration bias,

I. Issa and M. Gastpar, “Computable bounds on the exploration bias,” inIEEE International Symposium on Information Theory (ISIT). IEEE, 2018, pp. 576–580

work page 2018
[33]

Hypothesis testing under maximal leakage privacy constraints,

J. Liao, L. Sankar, F. P. Calmon, and V . Y . Tan, “Hypothesis testing under maximal leakage privacy constraints,” inIEEE International Symposium on Information Theory (ISIT). IEEE, 2017, pp. 779–783

work page 2017
[34]

A tunable measure for information leakage,

J. Liao, O. Kosut, L. Sankar, and F. P. Calmon, “A tunable measure for information leakage,” inIEEE International Symposium on Information Theory (ISIT). IEEE, 2018, pp. 701–705

work page 2018
[35]

Pointwise maximal leakage,

S. Saeidian, G. Cervia, T. J. Oechtering, and M. Skoglund, “Pointwise maximal leakage,”IEEE Transactions on Information Theory, vol. 69, no. 12, pp. 8054–8080, 2023

work page 2023
[36]

Generalization error bounds for noisy, iterative algorithms via maximal leakage,

I. Issa, A. R. Esposito, and M. Gastpar, “Generalization error bounds for noisy, iterative algorithms via maximal leakage,” inConference on Learning Theory (COLT). PMLR, 2023, pp. 4952–4976

work page 2023
[37]

Sibsonα-mutual information and its variational representations,

A. R. Esposito, M. Gastpar, and I. Issa, “Sibsonα-mutual information and its variational representations,”IEEE Transactions on Information Theory, vol. 72, no. 7, pp. 1–36, 2025

work page 2025
[38]

Tighter risk certificates for neural networks,

M. Pérez-Ortiz, O. Rivasplata, J. Shawe-Taylor, and C. Szepesvári, “Tighter risk certificates for neural networks,”Journal of Machine Learning Research, vol. 22, no. 227, pp. 1–40, 2021

work page 2021
[39]

Still no free lunches: the price to pay for tighter PAC-bayes bounds,

B. Guedj and L. Pujol, “Still no free lunches: the price to pay for tighter PAC-bayes bounds,”Entropy, vol. 23, no. 11, p. 1529, 2021

work page 2021
[40]

Non-vacuous generalisation bounds for shallow neural networks,

F. Biggs and B. Guedj, “Non-vacuous generalisation bounds for shallow neural networks,” inInternational conference on machine learning. PMLR, 2022, pp. 1963–1981

work page 2022
[41]

Some PAC-Bayesian theorems,

D. A. McAllester, “Some PAC-Bayesian theorems,” inConference on Computational Learning Theory, 1998, pp. 230–234

work page 1998
[42]

Pac-bayesian model averaging,

——, “Pac-bayesian model averaging,” inProceedings of the twelfth annual conference on Computational learning theory, 1999, pp. 164–170

work page 1999
[43]

Catoni,Statistical learning theory and stochastic optimization: Ecole d’Eté de Probabilités de Saint-Flour XXXI-2001

O. Catoni,Statistical learning theory and stochastic optimization: Ecole d’Eté de Probabilités de Saint-Flour XXXI-2001. Springer, 2004

work page 2001
[44]

Pac-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning

——, “PAC-Bayesian supervised classification: the thermodynamics of statistical learning,”arXiv preprint arXiv:0712.0248, 2007

work page internal anchor Pith review Pith/arXiv arXiv 2007
[45]

User-friendly introduction to pac-bayes bounds,

P. Alquier, “User-friendly introduction to pac-bayes bounds,”arXiv preprint arXiv:2110.11216, 2021

work page arXiv 2021
[46]

Generalization bounds via conditionalf-information,

Z. Wang and Y . Mao, “Generalization bounds via conditionalf-information,”Advances in Neural Information Processing Systems, vol. 37, pp. 52 159–52 188, 2024

work page 2024
[47]

Tighter CMI-based generalization bounds via stochastic projection and quantization,

M. Sefidgaran, K. Nadjahi, and A. Zaidi, “Tighter CMI-based generalization bounds via stochastic projection and quantization,”Advances in Neural Information Processing Systems, vol. 38, 2025

work page 2025
[48]

The algorithmic foundations of differential privacy,

C. Dwork, A. Rothet al., “The algorithmic foundations of differential privacy,”Foundations and Trends® in Theoretical Computer Science, vol. 9, no. 3–4, pp. 211–407, 2014

work page 2014
[49]

Upper bounds on the generalization error of private algorithms for discrete data,

B. Rodríguez-Gálvez, G. Bassi, and M. Skoglund, “Upper bounds on the generalization error of private algorithms for discrete data,”IEEE Transactions on Information Theory, vol. 67, no. 11, pp. 7362–7379, 2021

work page 2021
[50]

On the Generalization Error of Differentially Private Algorithms via Typicality

Y . Liu, C. H. M. Shiu, L. Wang, and D. Gündüz, “On the generalization error of differentially private algorithms via typicality,”arXiv preprint arXiv:2601.08386, 2026

work page internal anchor Pith review Pith/arXiv arXiv 2026
[51]

Algorithmic stability for adaptive data analysis,

R. Bassily, K. Nissim, A. Smith, T. Steinke, U. Stemmer, and J. Ullman, “Algorithmic stability for adaptive data analysis,” inProceedings of the forty-eighth annual ACM symposium on Theory of Computing, 2016, pp. 1046–1059

work page 2016
[52]

Information-theoretic generalization bounds for deep neural networks,

H. He and Z. Goldfeld, “Information-theoretic generalization bounds for deep neural networks,”IEEE Transactions on Information Theory, vol. 71, no. 8, pp. 6227–6247, 2025

work page 2025
[53]

Trade-offs in data memorization via strong data processing inequalities,

V . Feldman, G. Kornowski, and X. Lyu, “Trade-offs in data memorization via strong data processing inequalities,”arXiv preprint arXiv:2506.01855, 2025

work page arXiv 2025
[54]

A martingale approach to continuous-time marginal structural models,

K. Røysland, “A martingale approach to continuous-time marginal structural models,”Bernoulli, vol. 17, no. 3, pp. 895–915, Aug. 2011. February 11, 2026 DRAFT 17

work page 2011
[55]

Scalable information inequalities for uncertainty quantification,

M. A. Katsoulakis, L. Rey-Bellet, and J. Wang, “Scalable information inequalities for uncertainty quantification,”Journal of Computational Physics, vol. 336, pp. 513–545, 2017

work page 2017
[56]

Relating data compression and learnability,

N. Littlestone and M. Warmuth, “Relating data compression and learnability,” 1986

work page 1986
[57]

Statistical learning theory,

V . N. Vapnik, V . Vapniket al., “Statistical learning theory,” 1998

work page 1998
[58]

Theory of classification: A survey of some recent advances,

S. Boucheron, O. Bousquet, and G. Lugosi, “Theory of classification: A survey of some recent advances,”ESAIM: Probability and Statistics, vol. 9, pp. 323–375, 2005

work page 2005
[59]

Shalev-Shwartz and S

S. Shalev-Shwartz and S. Ben-David,Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014

work page 2014
[60]

Understanding deep learning requires rethinking generalization,

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” inInternational Conference on Learning Representations (ICLR), 2017

work page 2017
[61]

Stability and generalization,

O. Bousquet and A. Elisseeff, “Stability and generalization,”The Journal of Machine Learning Research, vol. 2, pp. 499–526, 2002

work page 2002
[62]

On the uniform convergence of relative frequencies of events to their probabilities,

V . N. Vapnik and A. Y . Chervonenkis, “On the uniform convergence of relative frequencies of events to their probabilities,” inMeasures of Complexity: Festschrift for Alexey Chervonenkis. Springer, 2015, pp. 11–30

work page 2015
[63]

Occam’s razor,

A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, “Occam’s razor,”Information processing letters, vol. 24, no. 6, pp. 377–380, 1987

work page 1987
[64]

Information complexity and generalization bounds,

P. K. Banerjee and G. Montúfar, “Information complexity and generalization bounds,” inIEEE International Symposium on Information Theory (ISIT). IEEE, 2021, pp. 676–681

work page 2021
[65]

Information-theoretic generalization bounds for black-box learning algorithms,

H. Harutyunyan, M. Raginsky, G. Ver Steeg, and A. Galstyan, “Information-theoretic generalization bounds for black-box learning algorithms,”Advances in Neural Information Processing Systems, vol. 34, pp. 24 670–24 682, 2021

work page 2021
[66]

Tightening mutual information-based bounds on generalization error,

Y . Bu, S. Zou, and V . V . Veeravalli, “Tightening mutual information-based bounds on generalization error,”IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 1, pp. 121–130, 2020

work page 2020
[67]

Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms,

M. Haghifam, J. Negrea, A. Khisti, D. M. Roy, and G. K. Dziugaite, “Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms,”Advances in Neural Information Processing Systems, vol. 33, pp. 9925– 9935, 2020

work page 2020
[68]

Towards a unified information-theoretic framework for generalization,

M. Haghifam, G. K. Dziugaite, S. Moran, and D. Roy, “Towards a unified information-theoretic framework for generalization,”Advances in Neural Information Processing Systems, vol. 34, pp. 26 370–26 381, 2021

work page 2021
[69]

Data-dependent generalization bounds via variable-size compressibility,

M. Sefidgaran and A. Zaidi, “Data-dependent generalization bounds via variable-size compressibility,”IEEE Transactions on Information Theory, 2024

work page 2024
[70]

Tighter expected generalization error bounds via convexity of information measures,

G. Aminian, Y . Bu, G. W. Wornell, and M. R. Rodrigues, “Tighter expected generalization error bounds via convexity of information measures,” inIEEE International Symposium on Information Theory (ISIT). IEEE, 2022, pp. 2481–2486

work page 2022
[71]

A generalization of the rate-distortion theory and applications,

M. Zakai and J. Ziv, “A generalization of the rate-distortion theory and applications,” inInformation Theory New Trends and Open Problems. Springer, 1975, pp. 87–123

work page 1975
[72]

Interpretations of rényi entropies and divergences,

P. Harremoës, “Interpretations of rényi entropies and divergences,”Physica A: Statistical Mechanics and its Applications, vol. 365, no. 1, pp. 57–62, 2006

work page 2006
[73]

P. D. Grünwald,The minimum description length principle. MIT press, 2007

work page 2007
[74]

Generalized cutoff rates and Rényi’s information measures,

I. Csiszár, “Generalized cutoff rates and Rényi’s information measures,”IEEE Transactions on Information Theory, vol. 41, no. 1, pp. 26–34, 2002

work page 2002
[75]

Rényi divergence and kullback-leibler divergence,

T. Van Erven and P. Harremos, “Rényi divergence and kullback-leibler divergence,”IEEE Transactions on Information Theory, vol. 60, no. 7, pp. 3797–3820, 2014

work page 2014
[76]

Generalization error bounds for noisy, iterative algorithms,

A. Pensia, V . Jog, and P. Loh, “Generalization error bounds for noisy, iterative algorithms,” inIEEE International Symposium on Information Theory (ISIT). IEEE, 2018, pp. 546–550

work page 2018
[77]

Gaussian differential privacy,

J. Dong, A. Roth, and W. J. Su, “Gaussian differential privacy,”Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 84, no. 1, pp. 3–37, 2022

work page 2022
[78]

Fundamental bound on the reliability of quantum information transmission,

N. Sharma and N. A. Warsi, “Fundamental bound on the reliability of quantum information transmission,”Physical Review Letters, vol. 110, no. 8, p. 080501, 2013

work page 2013
[79]

On the strong converses for the quantum channel capacity theorems

——, “On the strong converses for the quantum channel capacity theorems,”arXiv preprint arXiv:1205.1712, 2012

work page internal anchor Pith review Pith/arXiv arXiv 2012
[80]

Channel coding rate in the finite blocklength regime,

Y . Polyanskiy, H. V . Poor, and S. Verdú, “Channel coding rate in the finite blocklength regime,”IEEE Transactions on Information Theory, vol. 56, no. 5, pp. 2307–2359, 2010

work page 2010

Showing first 80 references.