{"id":"e676cea1-7b19-4771-a0dd-3a2ae3300800","arxiv_id":"2504.18455","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Multi-view representation learning generalization is bounded by the MDL of latent variables, and a Gaussian product mixture regularizer built from these bounds improves test accuracy over VIB baselines.","lead":"New theory bounds the generalization gap of distributed multi-view representation learning by the minimum description length of the learned representations relative to a data-dependent symmetric prior. The authors turn this into a Gaussian product mixture regularizer that outperforms VIB and CDVIB baselines on several image classification tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader's weakest assumption—that the learned priors do not provably satisfy the exact symmetry condition in the multi-view case—is not supported by the algorithm's structure. The prior is a product over samples of per-label Gaussian mixtures, which is exactly symmetric under label-preserving permutations of the latent variables, regardless of how the mixture parameters were estimated. The deferred 'almost symmetric' relaxation in the companion paper is therefore not required for the central theorems to apply. The lossy extension of Theorem 3 is present in Appendix A.1, and the practical approximations do not invalidate the theoretical motivation. Remaining caveats (approximate KL estimates, best-over-regularization-parameter reporting) are standard and do not rise to a load-bearing objection. Hence no significant concern is identified, and the reader's CONDITIONAL verdict need not change on this basis.","tokens_in":39398,"tokens_out":40939,"duration_ms":411009,"concrete_test":"Verify exact symmetry of the learned product prior numerically: train a small GPM-MDL multi-view model, fix a mini-batch of latent samples, and evaluate log Q(U|S,S',W_e) versus log Q(U_π|S,S',W_e) for several label-preserving permutations π (e.g., swapping same-label training/test samples). If the ratio equals 1 to machine precision, the reader's symmetry concern is settled and the stated bounds apply to the learned prior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"After scrutiny, the main theorems appear sound and the reader's symmetry concern does not land. The learned product Gaussian-mixture prior Q(U,U'|S,S',W_e)=∏_i Q_{Y_i}(U_i)Q_{Y_i'}(U_i') is exactly symmetric under label-preserving permutations of the latent coordinates: for each label c, the mixture Q_c is a single distribution shared by all samples of that label, so permuting the U's among same-label samples merely reorders identical factors. Training-order dependence of the EM updates affects which Q_c is produced for a given S, but for every fixed S the conditional prior is invariant in the sense of Definition 1. The lossy regularizer used in experiments is supported by the lossy extension in Appendix A.1, and the remaining approximations (Hershey-Olsen KL estimates, dropping the test half of MDL) are justified by symmetry and by the fact that only the relative magnitude matters for a regularizer. Footnote 1's deferral to a single-view 'almost symmetric' result is a robustness statement, not a needed patch for the actual prior. Thus the central claim—that MDL-based bounds for symmetric priors motivate and are consistent with the proposed GPM-MDL regularizer—holds.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies distributed multi-view representation learning where K agents each observe a distinct view and independently produce a latent representation, and a decoder predicts a shared label from the K representations. The main theoretical contribution is a set of generalization bounds, in expectation and in probability, expressed in terms of the minimum description length (MDL) of the latent variables, defined as the KL divergence between the representation distribution on training/test data and a data-dependent symmetric prior. A further decomposition (Theorem 4) separates MDL into per-view marginal MDL terms minus a joint KL term, which is interpreted as showing that statistically redundant representations are favorable for generalization. The paper then proposes data-dependent Gaussian mixture priors and a Gaussians-product mixture prior for the multi-view setting, leading to the GM-MDL and GPM-MDL regularizers, and reports experiments on image classification benchmarks showing gains over no regularization, VIB, and CDVIB.","tokens_in":39591,"tokens_out":28159,"duration_ms":294660,"significance":"If the bounds are correct, the paper provides a principled MDL-based alternative to mutual-information-based IB regularizers for multi-view representation learning, with a concrete theoretical mechanism for the role of redundancy across views. The proof strategy based on Donsker-Varadhan change of measure and MGF bounding is concrete, and the multi-view decomposition in Theorem 4 is a genuinely useful conceptual step. The experimental study is broad (single-view and multi-view, several datasets, several encoder architectures, K up to 8) and the code is provided, which strengthens the empirical contribution. However, the formal support for the lossy regularizer actually used in the experiments is incomplete, and several symmetry-related statements in the manuscript are inconsistent and need clarification before the application claims are fully supported.","major_comments":[{"comment":"The lossy bound (17) is stated as an extension of Theorem 2, but Appendix A.1 derives a lossy version of Theorem 3 (Eqs. (34)-(37)) and does not prove (17). More importantly, the lossy regularizer used in the experiments (Section 4.1.2, Eqs. (27)-(28), and Appendix B.3) is not shown to correspond to a quantized encoder W_hat satisfying the distortion criterion in (17). The Gaussian-perturbation construction in Appendix B.3 defines a KL-type regularizer, but no proof is given that this regularizer is the MDL term of any quantized model covered by the stated lossy bound. The authors should either provide the missing derivation connecting the lossy regularizer to the lossy bounds, or explicitly state that the lossy regularizer is a heuristic motivated by, but not certified by, the theorems.","section":"Section 3.3 and Appendix A.1; Section 4.1.2 and Appendix B.3"},{"comment":"The manuscript says that the learned priors satisfy the symmetry condition of Definition 1 only partially and defers formal 'almost symmetric' results to a companion paper, with formal results available only for the single-view setup. For the proposed factorized prior Q(U,U'|S,S',W_e) = ∏_i Q_{Y_i}(U_i)Q_{Y_i'}(U_i'), however, the same distribution Q_c is used for every sample of label c, so for any fixed S the prior appears to be exactly invariant under label-preserving permutations of the latent coordinates. The paper should resolve this inconsistency: if the prior is exactly symmetric, remove the caveat and the multi-view gap disappears; if some aspect of the online updates or the lossy construction breaks exact symmetry, the multi-view version of the almost-symmetric relaxation needs to be supplied before the bounds can be applied to the proposed algorithm.","section":"Footnote 1 and Section 4.1.1"},{"comment":"The proof of Theorem 4 constructs a prior P in Eq. (12) and states that it satisfies the symmetry property with respect to the particular permutation π_{Y,Y'}, while Theorem 2 as stated (with Definition 1) requires invariance under all label-preserving permutations. The proof then asserts, without proof, that the symmetry requirement of Theorem 2 can be weakened to this single permutation. This is a load-bearing step: if the weaker condition is not sufficient, the derivation of the bound MDL(P) ≤ MDLdist does not go through. The authors should either prove the weakened-symmetry claim or prove that P is fully symmetric under Definition 1 (which appears to follow from exchangeability of X,X' given Y,Y'); as written, the proof is incomplete.","section":"Appendix D.2, proof of Theorem 4"}],"minor_comments":[{"comment":"The claim that the residual term B_emp_diff is negligible because ||p_Y - p_Y'||_1 is of order sqrt(C/n) is only supported by a numerical illustration (Figure 2). The function h_C is not uniformly Lipschitz on [0,1]^2, so an analytical bound or a clear heuristic qualification would be preferable.","section":"Section 3.1, Eq. (11) and surrounding text"},{"comment":"There are several typos and notation issues: 'nultiterminal' should be 'multiterminal', 'orrder' should be 'order', 'satisfing' should be 'satisfying', 'Theorme' in Appendix D.1 should be 'Theorem', and 'Xβ' in Section 4.2.2 should be 'X_B'. The Markov chain in Section 3.1 is written with 'Ø' symbols and should use arrows (U1 -> X1 -> Y -> X2 -> U2).","section":"Throughout"},{"comment":"The comparison with Aguerri and Zaidi's complexity term is informal: the Markov-chain condition and the equivalence between the two forms of R1+R2 should be stated more carefully, and it should be made explicit that the comparison is an analogy between different theoretical frameworks rather than a direct implication.","section":"Section 3.1, Eqs. (14)-(15)"},{"comment":"All multi-view results are reported as averages over 5 runs without error bars or standard deviations; adding variance information would make the comparisons more convincing, especially where gains are small (e.g., Table 3, several rows).","section":"Section 5 and Table 3"},{"comment":"The joint regularizer involves a sum over M^K mixture components, which can be large for K up to 8. The paper does not discuss the computational cost or how M is chosen in the experiments; a brief complexity note would be helpful.","section":"Section 4.2.2, Eqs. (31)-(32)"}],"recommendation":"major_revision","confidential_remarks":"The paper leans heavily on the authors' own prior work (Sefidgaran et al., 2023, 2025) for key proof components and for the 'almost symmetric' relaxation. For a journal submission, the editor may wish to ask the authors to make the present paper more self-contained, particularly for Theorem 4's symmetry argument and the lossy regularizer connection. The central theoretical ideas appear sound, but the application claims in their current form outrun the formal results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the authors' own earlier MDL bound extended to multi-view learning, with three genuinely new theorems and a practical regularizer that wins consistently in their experiments. It deserves a serious referee. The concern you heard about the learned prior not satisfying the symmetry condition is, on reading, not a real problem: the product-of-mixtures prior is exactly symmetric under label-preserving permutations for any fixed data and encoder, since permuting same-label samples just reorders identical mixture factors. Footnote 1's 'almost symmetric' remark is a robustness aside, not a patch for this specific prior.\n\nThe genuinely new content is Theorem 3 (an MDL/n in-expectation bound using the hD and label-discrepancy machinery), Theorem 4 (decomposition of MDL into marginal MDLs minus a joint KL term, giving the redundancy-is-helpful read), and Theorem 5 (a tail bound). The proofs look careful: Donsker-Varadhan plus an MGF argument, with the hC residual kept explicit. The lossy extension in Appendix A.1 is a reasonable way to avoid vacuous bounds for deterministic encoders. The regularizer itself is a sensible use of the bound: a per-label Gaussian product mixture, with distributed updates and a natural attention-like coefficient. The code is public, which helps.\n\nWhere I'd push back: the route from the bound to the actual regularizer is heuristic, not a guarantee. They approximate the mixture KL with Hershey-Olsen, drop the test half of the MDL, and switch to a lossy version; all are practical and clearly stated, but the stated theorem is not exactly what the regularizer optimizes. The experiments are reported as best-over-regularization-parameter accuracy, averaged over 5 runs, without error bars. The gains are consistent and often a few points, but the oracle selection and lack of variance make the size of the effect hard to judge. And the 'redundant features help' conclusion is a bound-improvement result: a smaller upper bound, not a proven property of true risk. That's fine, but worth being explicit.\n\nWho should read this: people working on PAC-Bayes and compression bounds for representation learning, and anyone building distributed multi-view regularizers. It is a solid subfield contribution, not a breakthrough. I'd send it to a serious referee; if I were the editor, I'd ask for the experimental reporting to be tightened (error bars, and a fixed or validation-selected regularization parameter) and for a clean statement of which bounds apply to the actual regularizer.","headline":"Solid extension of the authors' MDL bounds to multi-view learning with a practical regularizer and consistent gains; the symmetry concern does not land, but the theory-to-practice link and the oracle-style experiments are the real soft spots.","tokens_in":40199,"tokens_out":5418,"would_cite":true,"duration_ms":56563,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68Q32","68T05","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"For distributed multi-view representation learning, the paper establishes that the generalization gap is controlled by the minimum description length of the latent code against a symmetric prior, with statistically redundant views…","keywords":["multi-view representation learning","generalization bounds","minimum description length","Gaussian product mixture prior","regularization","redundant representations","information bottleneck","distributed learning"],"falsifier":"Re-run the two-view CIFAR10 experiments of Table 3 while measuring, on every training checkpoint, the actual symmetry violation of the online-learned prior (the change in prior mass under a label-preserving permutation of the latent codes) and all terms of the Theorem 3 bound, then compare the bound with the realized test-minus-train risk across the five seeds: if the inequality fails by more than the $\\sqrt{C/n}$-sized residual, or if increasing the measured joint-KL term (more redundant representations) is accompanied by a strictly larger generalization gap, the central claim is false in exactly the regime the paper applies it.","tokens_in":39127,"feed_emoji":"🧩","tokens_out":19152,"duration_ms":167256,"temperature":0.7,"pith_summary":"This paper asks what each of $K$ independently operating encoders must extract from its own view of the data so that a central decoder, seeing all $K$ representations, predicts the label correctly on new data. Its answer, from a generalization-theory perspective, is that the decisive quantity is not the mutual information between inputs and representations — the quantity information-bottleneck methods regularize — but the minimum description length of the latent code: the expected relative entropy between the representations produced on training and test data and a symmetric, data-dependent prior. The paper proves generalization bounds of this form, in expectation and with high probability, the tightest decaying roughly as $\\mathrm{MDL}(Q)/n$, and proves that this description length splits into per-view marginal terms minus a joint relative-entropy term, so statistically redundant (correlated) representations among views shrink the bound. It then turns the bound into a regularizer built from Gaussian mixture priors updated online; a weighted attention mechanism emerges from the update rule, and experiments on CIFAR10, CIFAR100, USPS, and INTEL show the regularizer outperforming variational information bottleneck baselines in both single-view and multi-view settings.","feed_headline":"Compression, not mutual information, governs multi-view learning","feed_subtitle":"New bounds tie generalization to the description length of latent codes; correlated views shrink the bound","key_machinery":"Three objects carry the argument. (1) The symmetric prior (Definition 1): a conditional distribution $Q$ over the $2n$ latent codes that is invariant under every permutation of samples preserving the labels; the theorems hold for any such $Q$, and this invariance is what licenses the reshuffling of latent codes among same-label samples in the proof. (2) The MDL functional $\\mathrm{MDL}(Q)$: the expected relative entropy between the representation distributions produced by the stochastic encoders on the training and ghost sets and $Q$; this is the complexity measure the regularizer penalizes, and the bound's dominant term is $\\mathrm{MDL}(Q)/n$. (3) The decomposition of Theorem 4: one admissible prior makes the joint MDL equal to the sum of per-view marginal MDLs minus the relative entropy between the joint representation distribution and the product of per-view priors, and the negative sign of this joint term is the formal reason redundant views shrink the bound. On the algorithmic side, the machinery is a per-class Gaussian product mixture prior, updated with an EM-like procedure in which the KL divergence to the mixture is approximated by the average of a variational lower bound and a product-Gaussian upper bound, and whose membership weights $\\gamma_{i,m}$ reduce to a weighted attention rule when the lossy objective is used.","core_discovery":"The central claim is that for any symmetric conditional prior $Q$ — invariant under permutations of the $2n$ training and ghost-test samples that preserve labels — the expected generalization gap of a $K$-view representation learning algorithm is bounded by $(\\mathrm{MDL}(Q)+\\log n)/n$ plus a residual that involves only the empirical label distributions of the two sample sets, where $\\mathrm{MDL}(Q)$ is the expected relative entropy between the posterior distribution of the latent codes given the views and $Q$; the gap is measured through a binary-entropy function $h_D$ that upper-bounds both the expected 0-1 risk difference and its square. The paper further shows that one admissible choice of $Q$ splits this MDL into the sum of per-view marginal MDL terms minus a joint relative-entropy term, and because that joint term enters negatively, the more the views' representations agree statistically, the smaller the bound. From this the authors conclude that distributed encoders should be steered to emit redundant features, and that the description length of the latent code — not its mutual information with the input — is the quantity a regularizer should penalize. They instantiate the idea with class-conditional Gaussian mixture priors over the joint latent space, updated online so that most computation stays at the clients, and a lossy variant whose mixture-membership weights update through a formula that is structurally a weighted attention mechanism.","pith_inferences":["The learned membership weights $\\gamma_{i,m}$ provide a ready-made per-view relevance score: a view whose marginal MDL contribution is large relative to its share of the joint term is a candidate for pruning, a view-selection protocol the paper does not develop.","Because the regularizer touches only the encoders, the approach carries over to settings with a privacy boundary between clients and server: the distributed update protocol already keeps per-view statistics local, so the joint mixture could be aggregated without ever sharing raw representations or full latent codes.","The redundancy conclusion is a statement about the generalization bound; whether test accuracy itself is monotone in the joint term is an empirical question the paper's experiments do not isolate, and a natural follow-up is a controlled sweep in which the same data is encoded with deliberately correlated versus decorrelated representations."],"forward_implications":["If the bounds hold, a multi-view system's generalization guarantee depends on the encoders alone — the decoder's complexity drops out entirely — so MDL of the latent code is a legitimate standalone regularizer, replacing mutual-information-based information bottleneck penalties.","The Theorem 4 decomposition makes redundancy beneficial for generalization: adding statistically correlated structure across views lowers the joint term and tightens the bound, while a marginals-only regularizer that penalizes the same structure twice works against the theory.","The lossy variant of the bound stays non-vacuous for deterministic encoders and offers a quantitative handle on the empirically observed 'geometrical compression' of learned representations, something mutual-information bounds cannot do.","In the reported experiments the GM-MDL regularizer improves test accuracy over VIB and CDVIB baselines in the single-view setting and over per-view VIB and no-regularization in settings with 2 to 8 views across CIFAR10, CIFAR100, USPS, and INTEL."],"supporting_citations":[{"why":"Its single-view MDL generalization theorem is the result Theorem 2 adapts to multiple views, and it supplies the binary-entropy functions used in the proofs plus the CDVIB baseline the single-view experiments beat.","marker":"Sefidgaran et al. 2023"},{"why":"Companion paper carrying the lossy Gaussian-mixture-prior machinery, the KL-estimation details, and the almost-symmetric prior relaxation the online priors rely on, proved there only for the single-view case.","marker":"Sefidgaran et al. 2025"},{"why":"Its change-of-measure lemma is the first step of the proofs of Theorems 3 and 5, converting the expected-gap statement into an exponential-moment bound.","marker":"Donsker and Varadhan 1975"},{"why":"Supplies the variational and product-Gaussian bounds whose average approximates the KL divergence to a Gaussian mixture inside the regularizer.","marker":"Hershey and Olsen 2007"},{"why":"Bounds the total variation between the empirical label distributions by a square-root-of-C-over-n term, the step that makes the residual in Theorem 3 negligible.","marker":"Berend and Kontorovich 2012"},{"why":"Defines the variational information bottleneck objective used as the single-view baseline and the one-latent-sample-per-image sampling protocol the experiments follow.","marker":"Alemi et al. 2017"},{"why":"Provides the per-view multi-view information bottleneck baseline that GPM-MDL is compared against in the multi-view experiments.","marker":"Wan et al. 2021"},{"why":"Its distributed information bottleneck complexity expression, written as two marginal mutual informations minus the mutual information between the two representations, is the contrast the authors use to argue the redundancy conclusion is shared.","marker":"Aguerri and Zaidi 2021"}],"fun_headline_variants":["Description length, not mutual info, sets multi-view learning bounds","Redundant features aid multi-view generalization, new theory shows","Gaussian mixture priors tighten multi-view generalization guarantees","MDL-based regularizer outperforms VIB and CDVIB in tests","Weighted attention emerges from MDL-driven multi-view learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every main theorem requires the prior to be exactly symmetric — invariant under any reshuffling of the samples that keeps the labels fixed — but the priors used in the experiments are updated online from the training data and are not guaranteed to satisfy that condition; the paper itself flags in a footnote to Section 4.1 that the relaxation to 'almost symmetric' priors is only formally proved for the single-view setup, not for the multi-view experiments.","fun_headline_variants_meta":{"raw":{"variants":["Description length, not mutual info, sets multi-view learning bounds","Redundant features aid multi-view generalization, new theory shows","Gaussian mixture priors tighten multi-view generalization guarantees","MDL-based regularizer outperforms VIB and CDVIB in tests","Weighted attention emerges from MDL-driven multi-view learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":2070,"prompt_tokens":1145,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":761,"completion_tokens_details":{"reasoning_tokens":840}},"tokens_in":761,"tokens_out":925,"duration_ms":9058,"temperature":1.0,"reasoning_tokens":840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:16:54.834180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two-view CIFAR10 experiments of Table 3 while measuring, on every training checkpoint, the actual symmetry violation of the online-learned prior (the change in prior mass under a label-preserving permutation of the latent codes) and all terms of the Theorem 3 bound, then compare the bound with the realized test-minus-train risk across the five seeds: if the inequality fails by more than the $\\sqrt{C/n}$-sized residual, or if increasing the measured joint-KL term (more redundant representations) is accompanied by a strictly larger generalization gap, the central claim is false in exactly the regime the paper applies it.","supporting_citations":[{"cited_title":"Minimum description length and generalization guarantees for representation learning","cited_arxiv_id":null,"evidence_quote":"Its single-view MDL generalization theorem is the result Theorem 2 adapts to multiple views, and it supplies the binary-entropy functions used in the proofs plus the CDVIB baseline the single-view experiments beat."},{"cited_title":"Generalization guarantees for representation learning via data-dependent gaussian mixture priors","cited_arxiv_id":null,"evidence_quote":"Companion paper carrying the lossy Gaussian-mixture-prior machinery, the KL-estimation details, and the almost-symmetric prior relaxation the online priors rely on, proved there only for the single-view case."},{"cited_title":"Approximating the kullback leibler divergence between gaussian mixture models","cited_arxiv_id":null,"evidence_quote":"Supplies the variational and product-Gaussian bounds whose average approximates the KL divergence to a Gaussian mixture inside the regularizer."}],"review_version":1}