Pith. sign in

REVIEW 3 major objections 5 minor 101 references

Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For distributed multi-view representation learning, the paper establishes that the generalization gap is controlled by the minimum description length of the latent code against a symmetric prior, with statistically redundant views…

desk verdict Solid extension of the authors' MDL bounds to multi-view learning with a practical regularizer and consistent gains; the symmetry concern does not land, but the theory-to-practice link and the oracle-style experiments are the real soft spots. read the letter →

arxiv 2504.18455 v1 pith:MAQXY44Q submitted 2025-04-25 stat.ML cs.ITcs.LGmath.IT

Milad Sefidgaran , Abdellatif Zaidi , Piotr Krasnowski This is my paper · ORCID
classification stat.MLcs.ITcs.LGmath.IT MSC 68Q3268T0594A17
keywords multi-viewrepresentationlearninggeneralizationboundsminimumdescriptionlengthGaussianproductmixturepriorregularizationredundantrepresentationsinformationbottleneckdistributed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what each of $K$ independently operating encoders must extract from its own view of the data so that a central decoder, seeing all $K$ representations, predicts the label correctly on new data. Its answer, from a generalization-theory perspective, is that the decisive quantity is not the mutual information between inputs and representations — the quantity information-bottleneck methods regularize — but the minimum description length of the latent code: the expected relative entropy between the representations produced on training and test data and a symmetric, data-dependent prior. The paper proves generalization bounds of this form, in expectation and with high probability, the tightest decaying roughly as $\mathrm{MDL}(Q)/n$, and proves that this description length splits into per-view marginal terms minus a joint relative-entropy term, so statistically redundant (correlated) representations among views shrink the bound. It then turns the bound into a regularizer built from Gaussian mixture priors updated online; a weighted attention mechanism emerges from the update rule, and experiments on CIFAR10, CIFAR100, USPS, and INTEL show the regularizer outperforming variational information bottleneck baselines in both single-view and multi-view settings.

What carries the argument

Three objects carry the argument. (1) The symmetric prior (Definition 1): a conditional distribution $Q$ over the $2n$ latent codes that is invariant under every permutation of samples preserving the labels; the theorems hold for any such $Q$, and this invariance is what licenses the reshuffling of latent codes among same-label samples in the proof. (2) The MDL functional $\mathrm{MDL}(Q)$: the expected relative entropy between the representation distributions produced by the stochastic encoders on the training and ghost sets and $Q$; this is the complexity measure the regularizer penalizes, and the bound's dominant term is $\mathrm{MDL}(Q)/n$. (3) The decomposition of Theorem 4: one admissible prior makes the joint MDL equal to the sum of per-view marginal MDLs minus the relative entropy between the joint representation distribution and the product of per-view priors, and the negative sign of this joint term is the formal reason redundant views shrink the bound. On the algorithmic side, the machinery is a per-class Gaussian product mixture prior, updated with an EM-like procedure in which the KL divergence to the mixture is approximated by the average of a variational lower bound and a product-Gaussian upper bound, and whose membership weights $\gamma_{i,m}$ reduce to a weighted attention rule when the lossy objective is used.

What would settle it

Re-run the two-view CIFAR10 experiments of Table 3 while measuring, on every training checkpoint, the actual symmetry violation of the online-learned prior (the change in prior mass under a label-preserving permutation of the latent codes) and all terms of the Theorem 3 bound, then compare the bound with the realized test-minus-train risk across the five seeds: if the inequality fails by more than the $\sqrt{C/n}$-sized residual, or if increasing the measured joint-KL term (more redundant representations) is accompanied by a strictly larger generalization gap, the central claim is false in exactly the regime the paper applies it.

Watch

Extended reading notes

Core claim

The central claim is that for any symmetric conditional prior $Q$ — invariant under permutations of the $2n$ training and ghost-test samples that preserve labels — the expected generalization gap of a $K$-view representation learning algorithm is bounded by $(\mathrm{MDL}(Q)+\log n)/n$ plus a residual that involves only the empirical label distributions of the two sample sets, where $\mathrm{MDL}(Q)$ is the expected relative entropy between the posterior distribution of the latent codes given the views and $Q$; the gap is measured through a binary-entropy function $h_D$ that upper-bounds both the expected 0-1 risk difference and its square. The paper further shows that one admissible choice of $Q$ splits this MDL into the sum of per-view marginal MDL terms minus a joint relative-entropy term, and because that joint term enters negatively, the more the views' representations agree statistically, the smaller the bound. From this the authors conclude that distributed encoders should be steered to emit redundant features, and that the description length of the latent code — not its mutual information with the input — is the quantity a regularizer should penalize. They instantiate the idea with class-conditional Gaussian mixture priors over the joint latent space, updated online so that most computation stays at the clients, and a lossy variant whose mixture-membership weights update through a formula that is structurally a weighted attention mechanism.

Load-bearing premise

Every main theorem requires the prior to be exactly symmetric — invariant under any reshuffling of the samples that keeps the labels fixed — but the priors used in the experiments are updated online from the training data and are not guaranteed to satisfy that condition; the paper itself flags in a footnote to Section 4.1 that the relaxation to 'almost symmetric' priors is only formally proved for the single-view setup, not for the multi-view experiments.

Editorial extensions

If this is right

  • If the bounds hold, a multi-view system's generalization guarantee depends on the encoders alone — the decoder's complexity drops out entirely — so MDL of the latent code is a legitimate standalone regularizer, replacing mutual-information-based information bottleneck penalties.
  • The Theorem 4 decomposition makes redundancy beneficial for generalization: adding statistically correlated structure across views lowers the joint term and tightens the bound, while a marginals-only regularizer that penalizes the same structure twice works against the theory.
  • The lossy variant of the bound stays non-vacuous for deterministic encoders and offers a quantitative handle on the empirically observed 'geometrical compression' of learned representations, something mutual-information bounds cannot do.
  • In the reported experiments the GM-MDL regularizer improves test accuracy over VIB and CDVIB baselines in the single-view setting and over per-view VIB and no-regularization in settings with 2 to 8 views across CIFAR10, CIFAR100, USPS, and INTEL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The learned membership weights $\gamma_{i,m}$ provide a ready-made per-view relevance score: a view whose marginal MDL contribution is large relative to its share of the joint term is a candidate for pruning, a view-selection protocol the paper does not develop.
  • Because the regularizer touches only the encoders, the approach carries over to settings with a privacy boundary between clients and server: the distributed update protocol already keeps per-view statistics local, so the joint mixture could be aggregated without ever sharing raw representations or full latent codes.
  • The redundancy conclusion is a statement about the generalization bound; whether test accuracy itself is monotone in the joint term is an empirical question the paper's experiments do not isolate, and a natural follow-up is a controlled sweep in which the same data is encoded with deliberately correlated versus decorrelated representations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies distributed multi-view representation learning where K agents each observe a distinct view and independently produce a latent representation, and a decoder predicts a shared label from the K representations. The main theoretical contribution is a set of generalization bounds, in expectation and in probability, expressed in terms of the minimum description length (MDL) of the latent variables, defined as the KL divergence between the representation distribution on training/test data and a data-dependent symmetric prior. A further decomposition (Theorem 4) separates MDL into per-view marginal MDL terms minus a joint KL term, which is interpreted as showing that statistically redundant representations are favorable for generalization. The paper then proposes data-dependent Gaussian mixture priors and a Gaussians-product mixture prior for the multi-view setting, leading to the GM-MDL and GPM-MDL regularizers, and reports experiments on image classification benchmarks showing gains over no regularization, VIB, and CDVIB.

Significance. If the bounds are correct, the paper provides a principled MDL-based alternative to mutual-information-based IB regularizers for multi-view representation learning, with a concrete theoretical mechanism for the role of redundancy across views. The proof strategy based on Donsker-Varadhan change of measure and MGF bounding is concrete, and the multi-view decomposition in Theorem 4 is a genuinely useful conceptual step. The experimental study is broad (single-view and multi-view, several datasets, several encoder architectures, K up to 8) and the code is provided, which strengthens the empirical contribution. However, the formal support for the lossy regularizer actually used in the experiments is incomplete, and several symmetry-related statements in the manuscript are inconsistent and need clarification before the application claims are fully supported.

major comments (3)
  1. [Section 3.3 and Appendix A.1; Section 4.1.2 and Appendix B.3] The lossy bound (17) is stated as an extension of Theorem 2, but Appendix A.1 derives a lossy version of Theorem 3 (Eqs. (34)-(37)) and does not prove (17). More importantly, the lossy regularizer used in the experiments (Section 4.1.2, Eqs. (27)-(28), and Appendix B.3) is not shown to correspond to a quantized encoder W_hat satisfying the distortion criterion in (17). The Gaussian-perturbation construction in Appendix B.3 defines a KL-type regularizer, but no proof is given that this regularizer is the MDL term of any quantized model covered by the stated lossy bound. The authors should either provide the missing derivation connecting the lossy regularizer to the lossy bounds, or explicitly state that the lossy regularizer is a heuristic motivated by, but not certified by, the theorems.
  2. [Footnote 1 and Section 4.1.1] The manuscript says that the learned priors satisfy the symmetry condition of Definition 1 only partially and defers formal 'almost symmetric' results to a companion paper, with formal results available only for the single-view setup. For the proposed factorized prior Q(U,U'|S,S',W_e) = ∏_i Q_{Y_i}(U_i)Q_{Y_i'}(U_i'), however, the same distribution Q_c is used for every sample of label c, so for any fixed S the prior appears to be exactly invariant under label-preserving permutations of the latent coordinates. The paper should resolve this inconsistency: if the prior is exactly symmetric, remove the caveat and the multi-view gap disappears; if some aspect of the online updates or the lossy construction breaks exact symmetry, the multi-view version of the almost-symmetric relaxation needs to be supplied before the bounds can be applied to the proposed algorithm.
  3. [Appendix D.2, proof of Theorem 4] The proof of Theorem 4 constructs a prior P in Eq. (12) and states that it satisfies the symmetry property with respect to the particular permutation π_{Y,Y'}, while Theorem 2 as stated (with Definition 1) requires invariance under all label-preserving permutations. The proof then asserts, without proof, that the symmetry requirement of Theorem 2 can be weakened to this single permutation. This is a load-bearing step: if the weaker condition is not sufficient, the derivation of the bound MDL(P) ≤ MDLdist does not go through. The authors should either prove the weakened-symmetry claim or prove that P is fully symmetric under Definition 1 (which appears to follow from exchangeability of X,X' given Y,Y'); as written, the proof is incomplete.
minor comments (5)
  1. [Section 3.1, Eq. (11) and surrounding text] The claim that the residual term B_emp_diff is negligible because ||p_Y - p_Y'||_1 is of order sqrt(C/n) is only supported by a numerical illustration (Figure 2). The function h_C is not uniformly Lipschitz on [0,1]^2, so an analytical bound or a clear heuristic qualification would be preferable.
  2. [Throughout] There are several typos and notation issues: 'nultiterminal' should be 'multiterminal', 'orrder' should be 'order', 'satisfing' should be 'satisfying', 'Theorme' in Appendix D.1 should be 'Theorem', and 'Xβ' in Section 4.2.2 should be 'X_B'. The Markov chain in Section 3.1 is written with 'Ø' symbols and should use arrows (U1 -> X1 -> Y -> X2 -> U2).
  3. [Section 3.1, Eqs. (14)-(15)] The comparison with Aguerri and Zaidi's complexity term is informal: the Markov-chain condition and the equivalence between the two forms of R1+R2 should be stated more carefully, and it should be made explicit that the comparison is an analogy between different theoretical frameworks rather than a direct implication.
  4. [Section 5 and Table 3] All multi-view results are reported as averages over 5 runs without error bars or standard deviations; adding variance information would make the comparisons more convincing, especially where gains are small (e.g., Table 3, several rows).
  5. [Section 4.2.2, Eqs. (31)-(32)] The joint regularizer involves a sum over M^K mixture components, which can be large for K up to 8. The paper does not discuss the computational cost or how M is chosen in the experiments; a brief complexity note would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified; bounds are derived inequalities and the regularizer is an application of them.

full rationale

The paper's central derivation is a PAC-Bayesian/change-of-measure argument: Theorem 3 bounds the expected generalization gap in terms of MDL(Q), which is itself defined as an expected KL divergence between the encoder-induced representation distribution and a chosen symmetric prior (Eq. 4). The regularizer used in experiments is then obtained by minimizing this same MDL term, so the regularizer is an application of the bound rather than an input that forces the bound. Theorem 4's redundancy conclusion follows algebraically from the decomposition MDLdist = sum of marginal MDLs minus a non-negative joint KL term (Eq. 13); it is a mathematical consequence of the construction, not a hidden restatement of the desired result. The main self-citations, notably the restatement of (Sefidgaran et al., 2023, Theorem 4) as Theorem 2 and the reuse of that paper's MGF argument in the proof of Theorem 3, cite previously proved theorems with independent content and are used as standard lemmas rather than as unverified assumptions that presuppose the present claim. The acknowledged gap that the practical online-learned prior may not satisfy the exact symmetry condition of Definition 1 is a soundness/rigor concern about applying the bounds to the implemented algorithm, not a circularity: the bounds themselves are stated for arbitrary symmetric priors. Experimental comparisons against VIB, CDVIB, and no-regularizer baselines further show that the empirical claims are tested against external alternatives rather than being determined by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The theorems are parameter-free in the mathematical sense (they hold for any fixed symmetric Q), but the regularizer introduces fitted Gaussian mixture parameters, hand-chosen hyperparameters, and lossy distortion terms. The key ad hoc axiom is the assumed almost-symmetric relaxation for the multi-view setup, which is load-bearing for the practical algorithm but deferred to a companion paper.

free parameters (4)
  • Gaussian mixture component parameters (means, variances, weights) = Per-class, per-component, per-view; learned during training
    The priors Q_c are Gaussian mixtures whose parameters mu_{c,m}, sigma_{c,m}, alpha_{c,m} are updated from mini-batch statistics via equations (24)-(25). The theorems hold for any fixed Q, so these are free parameters of the regularizer, not of the bounds.
  • Lossy distortion hyperparameter epsilon = Not specified (fixed constant)
    The lossy regularizer (28) adds epsilon to variances; the lossy bound (17) requires a distortion criterion with epsilon, but the paper does not quantify the achieved distortion.
  • Mixture component count M = Not specified in main text
    The number of Gaussian components per class is chosen by hand; larger M improves approximation capacity but adds parameters.
  • Update smoothing coefficients eta1, eta2, eta3 and noise scale zeta = Fixed constants in (25)
    Chosen by hand to reduce the dependence of the prior on the dataset and to partially preserve the symmetry property.
assumptions (5)
  • standard math Donsker-Varadhan variational inequality
    Used in the proofs of Theorems 3, 5, and the lossy bounds (Appendix D.1).
  • standard math Berend-Kontorovich bound on the total variation between empirical distributions
    Used to argue that the residual term B_emp_diff is small (Section 3.1).
  • domain assumption i.i.d. training and ghost samples; encoders are conditionally independent; representations are Gaussian with diagonal covariance
    The setup in Section 2 and equation (19); the posterior factorizes as product over samples and views of N(mu_{x_{i,k}}, diag(sigma^2_{x_{i,k}})).
  • ad hoc to paper Almost-symmetric prior relaxation holds for the multi-view setup with a small penalty
    The practical online-learned priors do not provably satisfy exact symmetry (Definition 1); the paper defers formal results for the single-view case to (Sefidgaran et al., 2025, Appendix B) and states 'it can be shown' for the multi-view case without a derivation (footnote page 13).
  • ad hoc to paper The lossy MDL bound applies at the operating point used in experiments
    The regularizer uses Dprod and Dvar approximations (44), (58) and drops the test-set part of MDL; the paper does not quantify the distortion epsilon achieved or the approximation error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior." pith.science (2026). https://pith.science/paper/MAQXY44Q

@misc{pith2026250418455,
  author       = {Pith},
  title        = {Pith review of: Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAQXY44Q}},
  note         = {Machine review of arXiv:2504.18455}
}
abstract

We study the problem of distributed multi-view representation learning. In this problem, $K$ agents observe each one distinct, possibly statistically correlated, view and independently extracts from it a suitable representation in a manner that a decoder that gets all $K$ representations estimates correctly the hidden label. In the absence of any explicit coordination between the agents, a central question is: what should each agent extract from its view that is necessary and sufficient for a correct estimation at the decoder? In this paper, we investigate this question from a generalization error perspective. First, we establish several generalization bounds in terms of the relative entropy between the distribution of the representations extracted from training and "test" datasets and a data-dependent symmetric prior, i.e., the Minimum Description Length (MDL) of the latent variables for all views and training and test datasets. Then, we use the obtained bounds to devise a regularizer; and investigate in depth the question of the selection of a suitable prior. In particular, we show and conduct experiments that illustrate that our data-dependent Gaussian mixture priors with judiciously chosen weights lead to good performance. For single-view settings (i.e., $K=1$), our experimental results are shown to outperform existing prior art Variational Information Bottleneck (VIB) and Category-Dependent VIB (CDVIB) approaches. Interestingly, we show that a weighted attention mechanism emerges naturally in this setting. Finally, for the multi-view setting, we show that the selection of the joint prior as a Gaussians product mixture induces a Gaussian mixture marginal prior for each marginal view and implicitly encourages the agents to extract and output redundant features, a finding which is somewhat counter-intuitive.

Figures

Figures reproduced from arXiv: 2504.18455 by the authors.

Figure 1
Figure 1. Distributed multi-view representa￾tion learning setup. The approach described thus far involves only a single encoder and a single decoder; and is sometimes loosely referred to as cen￾tralized representation learning, in reference to that all training data is available at one place, the encoder. In many real-world sce￾narios, however, multiple streams of data may be available each at a distinct encoder; every encode… view at source ↗
Figure 2
Figure 2. Values of hC ´ Lˆpy, ˆyq,Lˆpy 1 , ˆy 1 q; ϵ ¯ as function of the generalization error for the CIFAR10 dataset. error. The values are obtained for empirical risk of 0.05 and }pˆY ´ pˆY1}1 set to be of the order a C{n. As it is visible from the figure, the term inside the expectation of Bemp_diff is the order of magnitude smaller than the generalization error. This illustrates that even for settings with moderate data… view at source ↗
Figure 3
Figure 3. Comparison of the generalization bounds of Theorem 3 and Theorem 2 for the CIFAR10 dataset. Now, we state the result which is proved in Appendix D.2. Theorem 4. Consider the setup of Theorem 2. Let, for every k P rKs, Qk be some symmetric conditional distribution for the view Xk. Then, ES,W rgenpS, Wqs ď b 2 MDLdistpQ1,...,QKq`C`2 n , where MDLdistpQ1, . . . , QKq :“ ÿ kPrKs ESk,S1 k ,We,k ” DKL´ PUk,U1 k |Xk,X1 k ,… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

101 extracted references · 57 canonical work pages

  1. [1]

    Distributed variational representation learning

    Iñaki Estella Aguerri and Abdellatif Zaidi. Distributed variational representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 0 (1): 0 120--138, 2021. doi:10.1109/TPAMI.2019.2928806

    arXiv 2021

  2. [2]

    Alemi, Ian Fischer, Joshua V

    Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HyxQzBceg

    2017

  3. [3]

    User-friendly introduction to pac-bayes bounds

    Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv preprint arXiv:2110.11216, 2021

    arXiv 2021

  4. [4]

    An exact characterization of the generalization error for the gibbs algorithm

    Gholamali Aminian, Yuheng Bu, Laura Toni, Miguel Rodrigues, and Gregory Wornell. An exact characterization of the generalization error for the gibbs algorithm. Advances in Neural Information Processing Systems, 34: 0 8106--8118, 2021

    2021

  5. [5]

    Learning representations for neural network-based classification using the information bottleneck principle

    Rana Ali Amjad and Bernhard C Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, 42 0 (9): 0 2225--2239, 2019

    2019

  6. [6]

    Stronger generalization bounds for deep nets via a compression approach

    Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254--263. PMLR, 2018

    2018

  7. [7]

    K-means++: The advantages if careful seeding

    David Arthur. K-means++: The advantages if careful seeding. In Proc. Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 2007, pages 1027--1035, 2007

    2007

  8. [8]

    Heavy tails in SGD and compressibility of overparametrized neural networks

    Melih Barsbey, Milad Sefidgaran, Murat A Erdogdu, Ga \"e l Richard, and Umut S im s ekli. Heavy tails in SGD and compressibility of overparametrized neural networks. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021

    2021

Show all 101 references
  1. [9]

    Pac-bayesian bounds based on the r \'e nyi divergence

    Luc B \'e gin, Pascal Germain, Fran c ois Laviolette, and Jean-Francis Roy. Pac-bayesian bounds based on the r \'e nyi divergence. In Artificial Intelligence and Statistics, pages 435--444. PMLR, 2016

    2016

  2. [10]

    On the convergence of the empirical distribution

    Daniel Berend and Aryeh Kontorovich. On the convergence of the empirical distribution. arXiv preprint arXiv:1205.6711, 2012

    2012 arXiv

  3. [11]

    Intrinsic dimension, persistent homology and generalization in neural networks

    Tolga Birdal, Aaron Lou, Leonidas Guibas, and Umut S im s ekli. Intrinsic dimension, persistent homology and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  4. [12]

    Pac-mdl bounds

    Avrim Blum and John Langford. Pac-mdl bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pages 344--357. Springer, 2003

    2003

  5. [13]

    Occam's razor

    Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam's razor. Information processing letters, 24 0 (6): 0 377--380, 1987

    1987

  6. [14]

    Proper learning, helly number, and an optimal svm bound

    Olivier Bousquet, Steve Hanneke, Shay Moran, and Nikita Zhivotovskiy. Proper learning, helly number, and an optimal svm bound. In Conference on Learning Theory, pages 582--609. PMLR, 2020

    2020

  7. [15]

    Veeravalli

    Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening mutual information-based bounds on generalization error. IEEE Journal on Selected Areas in Information Theory, 1 0 (1): 0 121–130, May 2020. ISSN 2641-8770

    2020

  8. [16]

    A pac-bayesian approach to adaptive classification

    Olivier Catoni. A pac-bayesian approach to adaptive classification. preprint, 840, 2003

    2003

  9. [17]

    Learning with metric losses

    Dan Tsir Cohen and Aryeh Kontorovich. Learning with metric losses. In Conference on Learning Theory, pages 662--700. PMLR, 2022

    2022

  10. [18]

    A novel approach for effective multi-view clustering with information-theoretic perspective

    Chenhang Cui, Yazhou Ren, Jingyu Pu, Jiawei Li, Xiaorong Pu, Tianyi Wu, Yutao Shi, and Lifang He. A novel approach for effective multi-view clustering with information-theoretic perspective. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Approximating priors by mixtures of natural conjugate priors

    SR Dalal and WJ Hall. Approximating priors by mixtures of natural conjugate priors. Journal of the Royal Statistical Society: Series B (Methodological), 45 0 (2): 0 278--286, 1983

    1983

  12. [20]

    Asymptotic evaluation of certain markov process expectations for large time, i

    Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on pure and applied mathematics, 28 0 (1): 0 1--47, 1975

    1975

  13. [21]

    Learning optimal representations with the decodable information bottleneck

    Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. Advances in Neural Information Processing Systems, 33: 0 18674--18690, 2020

    2020

  14. [22]

    Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data

    Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017

    2017 arXiv

  15. [23]

    Data-dependent pac-bayes priors via differential privacy

    Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac-bayes priors via differential privacy. Advances in neural information processing systems, 31, 2018

    2018

  16. [24]

    Generalization error bounds via R \'enyi-, f -divergences and maximal leakage, 2020

    Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa. Generalization error bounds via R \'enyi-, f -divergences and maximal leakage, 2020

    2020

  17. [25]

    Distributed information bottleneck method for discrete and gaussian sources

    I \ n aki Estella Aguerri and Abdellatif Zaidi. Distributed information bottleneck method for discrete and gaussian sources. In International Zurich Seminar on Information and Communication (IZS 2018). Proceedings, pages 35--39. ETH Zurich, 2018

    2018

  18. [26]

    Learning robust representations via multi-view information bottleneck

    Marco Federici, Anjan Dutta, Patrick Forr \'e , Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020

    2002 arXiv

  19. [27]

    The conditional entropy bottleneck

    Ian Fischer. The conditional entropy bottleneck. Entropy, 22 0 (9): 0 999, 2020

    2020

  20. [28]

    On information plane analyses of neural network classifiers--a review

    Bernhard C Geiger. On information plane analyses of neural network classifiers--a review. IEEE Transactions on Neural Networks and Learning Systems, 2021

    2021

  21. [29]

    On the information dimension of stochastic processes

    Bernhard C Geiger and Tobias Koch. On the information dimension of stochastic processes. IEEE transactions on information theory, 65 0 (10): 0 6496--6518, 2019

    2019

  22. [30]

    Pac-bayesian learning of linear classifiers

    Pascal Germain, Alexandre Lacasse, Fran c ois Laviolette, and Mario Marchand. Pac-bayesian learning of linear classifiers. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 353--360, 2009

    2009

  23. [31]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  24. [32]

    Estimating information flow in deep neural networks

    Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference ...

    2019

  25. [33]

    Deep learning, 2016

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning, 2016

    2016

  26. [34]

    Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Daniel M. Roy. Towards a unified information-theoretic framework for generalization. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021

    2021

  27. [35]

    Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization

    Mahdi Haghifam, Borja Rodr \' guez-G \'a lvez, Ragnar Thobaben, Mikael Skoglund, Daniel M Roy, and Gintare Karolina Dziugaite. Limitations of information-theoretic generalization bounds for gradient descent methods in stochastic convex optimization. In International Conference...

    2023

  28. [36]

    A sharp lower bound for agnostic learning with sample compression schemes

    Steve Hanneke and Aryeh Kontorovich. A sharp lower bound for agnostic learning with sample compression schemes. In Algorithmic Learning Theory, pages 489--505. PMLR, 2019

    2019

  29. [37]

    Stable sample compression schemes: New applications and an optimal svm margin bound

    Steve Hanneke and Aryeh Kontorovich. Stable sample compression schemes: New applications and an optimal svm margin bound. In Algorithmic Learning Theory, pages 697--721. PMLR, 2021

    2021

  30. [38]

    Sample compression for real-valued learners

    Steve Hanneke, Aryeh Kontorovich, and Menachem Sadigurschi. Sample compression for real-valued learners. In Algorithmic Learning Theory, pages 466--488. PMLR, 2019

    2019

  31. [39]

    Universal bayes consistency in metric spaces

    Steve Hanneke, Aryeh Kontorovich, Sivan Sabato, and Roi Weiss. Universal bayes consistency in metric spaces. In 2020 Information Theory and Applications Workshop (ITA), pages 1--33. IEEE, 2020

    2020

  32. [40]

    Information-theoretic generalization bounds for black-box learning algorithms

    Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Information-theoretic generalization bounds for black-box learning algorithms. Advances in Neural Information Processing Systems, 34, 2021

    2021

  33. [41]

    A new family of generalization bounds using samplewise evaluated cmi

    Fredrik Hellstr \"o m and Giuseppe Durisi. A new family of generalization bounds using samplewise evaluated cmi. Advances in Neural Information Processing Systems, 35: 0 10108--10121, 2022

    2022

  34. [42]

    Approximating the kullback leibler divergence between gaussian mixture models

    John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07, volume 4, pages IV--317. IEEE, 2007

    2007

  35. [43]

    Generalization bounds using lower tail exponents in stochastic optimizers

    Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning, pages 8774--8795. PMLR, 2022

    2022

  36. [44]

    Generalization bounds via distillation

    Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. In International Conference on Learning Representations, 2021

    2021

  37. [45]

    A survey on information bottleneck

    Shizhe Hu, Zhengzheng Lou, Xiaoqiang Yan, and Yangdong Ye. A survey on information bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  38. [46]

    On the multi-view information bottleneck representation

    Teng-Hui Huang, Aly El Gamal, and Hesham El Gamal. On the multi-view information bottleneck representation. In 2022 IEEE Information Theory Workshop (ITW), pages 37--42. IEEE, 2022

    2022

  39. [47]

    Generalized information-theoretic multi-view clustering

    Weitian Huang, Sirui Yang, and Hongmin Cai. Generalized information-theoretic multi-view clustering. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16 0 (5): 0 550--554, 1994

    1994

  41. [49]

    Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Ma...

    2023

  42. [50]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015

    2015

  43. [51]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014

    2014

  44. [52]

    Gacs-korner common information variational autoencoder

    Michael Kleinman, Alessandro Achille, Stefano Soatto, and Jonathan Kao. Gacs-korner common information variational autoencoder. arXiv preprint arXiv:2205.12239, 2022

    2022 arXiv

  45. [53]

    Caveats for information bottleneck in deterministic scenarios

    Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. arXiv preprint arXiv:1808.07593, 2018

    2018 arXiv

  46. [54]

    Nonlinear information bottleneck

    Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. Entropy, 21 0 (12): 0 1181, 2019

    2019

  47. [55]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009

    2009

  48. [56]

    (not) bounding the true error

    John Langford and Rich Caruana. (not) bounding the true error. Advances in Neural Information Processing Systems, 14, 2001

    2001

  49. [57]

    Chaotic regularization and heavy-tailed limits for deterministic gradient descent

    Soon Hoe Lim, Yijun Wan, and Umut S im s ekli. Chaotic regularization and heavy-tailed limits for deterministic gradient descent. arXiv preprint arXiv:2205.11361, 2022

    2022 arXiv

  50. [58]

    Dual contrastive prediction for incomplete multi-view representation learning

    Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (4): 0 4447--4461, 2022

    2022

  51. [59]

    Relating data compression and learnability

    Nick Littlestone and Manfred Warmuth. Relating data compression and learnability. Citeseer, 1986

    1986

  52. [60]

    Information theoretic lower bounds for information theoretic upper bounds

    Roi Livni. Information theoretic lower bounds for information theoretic upper bounds. Advances in Neural Information Processing Systems, 36, 2023

    2023

  53. [61]

    Generalization bounds via convex analysis

    G \'a bor Lugosi and Gergely Neu. Generalization bounds via convex analysis. In Conference on Learning Theory, pages 3524--3546. PMLR, 2022

    2022

  54. [62]

    Recognizable information bottleneck

    Yilin Lyu, Xin Liu, Mingyang Song, Xinyue Wang, Yaxin Peng, Tieyong Zeng, and Liping Jing. Recognizable information bottleneck. arXiv preprint arXiv:2304.14618, 2023

    2023 arXiv

  55. [63]

    A note on the pac bayesian theorem

    Andreas Maurer. A note on the pac bayesian theorem. arXiv preprint cs/0411099, 2004

    2004 arXiv

  56. [64]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273--1282. PMLR, 2017

    2017

  57. [65]

    In-network learning for distributed training and inference in networks

    Matei Moldoveanu and Abdellatif Zaidi. In-network learning for distributed training and inference in networks. In 2021 IEEE Globecom Workshops (GC Wkshps), pages 1--6. IEEE, 2021

    2021

  58. [66]

    In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors

    Jeffrey Negrea, Gintare Karolina Dziugaite, and Daniel Roy. In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors. In International Conference on Machine Learning, pages 7263--7272. PMLR, 2020 a

    2020

  59. [67]

    Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M. Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates, 2020 b

    2020

  60. [68]

    Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M. Roy. Information-theoretic generalization bounds for stochastic gradient descent, 2021

    2021

  61. [69]

    A pac-bayesian approach to spectrally-normalized margin bounds for neural networks, 2018

    Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks, 2018

    2018

  62. [70]

    Improving transformers with probabilistic attention keys

    Tam Minh Nguyen, Tan Minh Nguyen, Dung DD Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving transformers with probabilistic attention keys. In International Conference on Machine Learning, pages 16595--16621. PMLR, 2022

    2022

  63. [71]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  64. [72]

    Tighter risk certificates for neural networks

    Mar \' a P \'e rez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesv \'a ri. Tighter risk certificates for neural networks. Journal of Machine Learning Research, 22 0 (227): 0 1--40, 2021

    2021

  65. [73]

    Pac-bayes analysis beyond the usual bounds

    Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesv \'a ri, and John Shawe-Taylor. Pac-bayes analysis beyond the usual bounds. Advances in Neural Information Processing Systems, 33: 0 16833--16845, 2020

    2020

  66. [74]

    The information bottleneck: Connections to other problems, learning and exploration of the ib curve, 2019

    Borja Rodriguez Galvez. The information bottleneck: Connections to other problems, learning and exploration of the ib curve, 2019

    2019

  67. [75]

    The convex information bottleneck lagrangian

    Borja Rodr \' guez G \'a lvez, Ragnar Thobaben, and Mikael Skoglund. The convex information bottleneck lagrangian. Entropy, 22 0 (1): 0 98, 2020

    2020

  68. [76]

    Controlling bias in adaptive data analysis using information theory

    Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine ...

    2016

  69. [77]

    Pac-bayesian generalisation error bounds for gaussian process classification

    Matthias Seeger. Pac-bayesian generalisation error bounds for gaussian process classification. Journal of machine learning research, 3 0 (Oct): 0 233--269, 2002

    2002

  70. [78]

    Data-dependent generalization bounds via variable-size compressibility

    Milad Sefidgaran and Abdellatif Zaidi. Data-dependent generalization bounds via variable-size compressibility. IEEE Transactions on Information Theory, 2024

    2024

  71. [79]

    Rate-distortion theoretic generalization bounds for stochastic learning algorithms

    Milad Sefidgaran, Amin Gohari, Gael Richard, and Umut Simsekli. Rate-distortion theoretic generalization bounds for stochastic learning algorithms. In Conference on Learning Theory, pages 4416--4463. PMLR, 2022

    2022

  72. [80]

    Minimum description length and generalization guarantees for representation learning

    Milad Sefidgaran, Abdellatif Zaidi, and Piotr Krasnowski. Minimum description length and generalization guarantees for representation learning. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  73. [81]

    Generalization guarantees for representation learning via data-dependent gaussian mixture priors

    Milad Sefidgaran, Abdellatif Zaidi, and Piotr Krasnowski. Generalization guarantees for representation learning via data-dependent gaussian mixture priors. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=fGdF8Bq1FV

    2025

  74. [82]

    Learning and generalization with the information bottleneck

    Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411 0 (29-30): 0 2696--2711, 2010

    2010

  75. [83]

    Hausdorff dimension, heavy tails, and generalization in neural networks

    Umut S im s ekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol...

    2020

  76. [84]

    R easoning about generalization via conditional mutual information

    Thomas Steinke and Lydia Zakynthinou. R easoning about generalization via conditional mutual information. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 3...

    2020

  77. [85]

    Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error

    Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning: Compressing deep neural networks via spectral analysis and its generalization error. In International Joint Conference o...

    2020

  78. [86]

    A strongly quasiconvex pac-bayesian bound

    Niklas Thiemann, Christian Igel, Olivier Wintenberger, and Yevgeny Seldin. A strongly quasiconvex pac-bayesian bound. In International Conference on Algorithmic Learning Theory, pages 466--492. PMLR, 2017

    2017

  79. [87]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000

    2000 arXiv

  80. [88]

    Pac-bayes-empirical-bernstein inequality

    Ilya O Tolstikhin and Yevgeny Seldin. Pac-bayes-empirical-bernstein inequality. Advances in Neural Information Processing Systems, 26, 2013

    2013

  81. [89]

    The role of the information bottleneck in representation learning

    Mat \' Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck in representation learning. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 1580--1584, 2018. doi:10.1109/ISIT.2018.8437679

    2018

  82. [90]

    A general framework for the disintegration of pac-bayesian bounds

    Paul Viallard, Pascal Germain, Amaury Habrard, and Emilie Morvant. A general framework for the disintegration of pac-bayesian bounds. arXiv preprint arXiv:2102.08649, 2021

    2021 arXiv

  83. [91]

    Multi-view information-bottleneck representation learning

    Zhibin Wan, Changqing Zhang, Pengfei Zhu, and Qinghua Hu. Multi-view information-bottleneck representation learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10085--10092, 2021

    2021

  84. [92]

    Cross-view representation learning for multi-view logo classification with information bottleneck

    Jing Wang, Yuanjie Zheng, Jingqi Song, and Sujuan Hou. Cross-view representation learning for multi-view logo classification with information bottleneck. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4680--4688, 2021

    2021

  85. [93]

    Deep multi-view information bottleneck

    Qi Wang, Claire Boudreau, Qixing Luo, Pang-Ning Tan, and Jiayu Zhou. Deep multi-view information bottleneck. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 37--45. SIAM, 2019

    2019

  86. [94]

    Information-theoretic analysis of generalization capability of learning algorithms

    Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 30, 2017

    2017

  87. [95]

    Deep multi-view learning methods: A review

    Xiaoqiang Yan, Shizhe Hu, Yiqiao Mao, Yangdong Ye, and Hui Yu. Deep multi-view learning methods: A review. Neurocomputing, 448: 0 106--129, 2021

    2021

  88. [96]

    Differentiable information bottleneck for deterministic multi-view clustering

    Xiaoqiang Yan, Zhixiang Jin, Fengshou Han, and Yangdong Ye. Differentiable information bottleneck for deterministic multi-view clustering. arXiv preprint arXiv:2403.15681, 2024

    2024 arXiv

  89. [97]

    On the information bottleneck problems: Models, connections, applications and information theoretic views

    Abdellatif Zaidi, I \ n aki Estella-Aguerri, and Shlomo Shamai. On the information bottleneck problems: Models, connections, applications and information theoretic views. Entropy, 22 0 (2): 0 151, 2020

    2020

  90. [98]

    Individually conditional individual mutual information bound on generalization error

    Ruida Zhou, Chao Tian, and Tie Liu. Individually conditional individual mutual information bound on generalization error. IEEE Transactions on Information Theory, 68 0 (5): 0 3304--3316, 2022. doi:10.1109/TIT.2022.3144615

    2022

  91. [99]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  92. [100]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  93. [101]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.