{"id":"671b1a9f-88dd-45ca-bca1-b2e6bf2e92a8","arxiv_id":"2505.01134","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CoDE-VAE computes the shared latent code of multiple data views by Bayesian combination of correlated per-view estimates, improving coherence/quality trade-offs and log-likelihood bounds.","lead":"A new Bayesian method for multimodal autoencoders treats each input type as an expert with correlated mistakes, instead of assuming the inputs are independent. On standard benchmarks it reports better balance between generated image quality and coherence, plus tighter log-likelihood estimates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may reflect test-set selection of ρ and β; held-out validation is needed before the empirical central claim is accepted.","rationale":"The reader's weakest_assumption was the joint-Gaussian, single-ρ model in Lemma 1, which is a legitimate modeling concern. However, the more load-bearing issue is the test-set selection of ρ and β, because it directly threatens the empirical support for the central claim. The paper explicitly admits selecting ρ for strongest overall performance and selecting PolyMNIST configurations by smallest FID, giving CoDE-VAE an uncontrolled advantage over baselines. Fixing the evaluation protocol is a prerequisite for accepting the empirical claims. The Appendix B positive-definiteness proof is indeed incorrect as written, but the specific parameterization σ_{i,j}=ρσ_iσ_j yields a rank-one-plus-diagonal covariance that is positive definite for ρ≥0, so this is repairable. The ELBO derivation in Appendix A.3 contains algebraic inconsistencies, but the final objective can be justified as a convex combination of per-subset ELBOs plus an entropy term and a constant, so it is not the primary threat. The availability of code is a positive signal and makes the proposed held-out validation check feasible. The verdict remains CONDITIONAL: accept after the selection protocol is corrected and the reported gains are confirmed on a properly held-out test set.","tokens_in":27004,"tokens_out":16536,"duration_ms":174794,"concrete_test":"Split the MNIST-SVHN-Text and PolyMNIST test sets into a validation split and a held-out test split. Select ρ from {0, 0.2, 0.4, 0.6, 0.8} and β from {0.1, 1, 5, 10, 15, 20} on the validation split using the same coherence/FID/log-likelihood criteria reported in the paper, then retrain CoDE-VAE at the selected values and report metrics on the held-out split. Apply the identical protocol to all baselines. If CoDE-VAE's advantage over PoE/MoE baselines shrinks to within one standard deviation, the empirical central claim does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that CoDE-VAE improves the coherence/quality trade-off and log-likelihood estimation over independent-expert baselines—rests on numbers obtained under a selection protocol that favors the proposed method. Section 4 states that 'all experiments present results that correspond to the ρ value demonstrating the strongest overall performance,' and ρ is exclusive to CoDE-VAE. Appendix D.4 further reports that for PolyMNIST, 120 configurations per modality count were trained and the architecture with the smallest FID was chosen. This is test-set selection of both the method-specific correlation parameter and the reported FID values. Baselines do not receive an equivalent test-set-selected free parameter; they use original implementations with their own optimal β, so the comparison is not controlled. If the improvement attributed to modeling dependence is partly an artifact of choosing ρ on the test set, the central claim is unsupported. The Gaussian/shared-ρ assumption flagged by the reader is a related but secondary concern: a single scalar ρ across all dimensions and expert pairs is restrictive, but the immediate threat to the paper's headline result is the evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoDE (Consensus of Dependent Experts), a Bayesian method for aggregating single-modality variational posteriors in multimodal VAEs. The key idea is to model the joint error of the expert estimates as multivariate Gaussian with a covariance that encodes dependence through a single scalar correlation parameter ρ, then compute the consensus posterior via a conjugate Gaussian update (Lemma 2). The paper also introduces a weighted ELBO (Lemma 3) in which each subset-specific ELBO term is weighted by a learned categorical coefficient π_k, aiming to improve the coherence/quality trade-off and log-likelihood estimation relative to standard PoE/MoE baselines. Experiments on MNIST-SVHN-Text, PolyMNIST, and CUB report better balance between generative coherence and quality, tighter log-likelihoods, and a reduced generative quality gap. The manuscript includes proofs in appendices, ablations, and released code.","tokens_in":27135,"tokens_out":9961,"duration_ms":96261,"significance":"If the results hold, the paper makes a useful contribution to multimodal VAE inference: it provides a principled alternative to PoE/MoE that explicitly accounts for dependence among experts, subsumes PoE as a special case, and addresses the equal-weight limitation of the generalized multimodal ELBO. The mathematical core of Lemma 2 is a standard and correct conjugate Gaussian calculation, and the release of code supports reproducibility. However, the empirical claims are currently undermined by the evaluation protocol, which selects the method-specific parameter ρ and the reported architecture on the test set, and the derivation of Lemma 3 has a mathematical gap. These issues must be resolved before the practical significance of the method can be assessed.","major_comments":[{"comment":"The reported empirical gains rely on test-set selection of the method-specific correlation parameter ρ and of the reported architecture. Section 4 states that 'all experiments present results that correspond to the ρ value demonstrating the strongest overall performance,' and Appendix D.4 reports that for PolyMNIST, 120 configurations per modality count were trained and 'the architecture with the smallest FID scores' was chosen. These selections are made on the same test set used for the headline numbers, and no equivalent test-set selection is applied to the baselines, which use their original implementations with their own optimal β. This protocol can only bias the comparison in favor of CoDE-VAE and does not support the central claim of an improved coherence/quality trade-off and tighter log-likelihoods. Please re-run the experiments using a held-out validation split to select ρ (and any architecture choices) and report results for the fixed validation-selected configuration.","section":"Section 4 and Appendix D.4"},{"comment":"The proof of Lemma 3 contains an invalid algebraic step. The derivation begins with the sum of KL divergences and obtains an expression involving (2^M−1) log p(X); it then writes '∝ Σ E[...] + log p(X)' by 'factor[ing] out 2^M−1 as a constant.' Since log p(X) is the quantity being bounded, dropping the factor (2^M−1) is not a benign proportionality step for an inequality. As a consequence, the subsequent line 'logp(X) ≥ Σ E[...]' does not follow from the preceding equations. The claimed lower bound in Eq. (3) may still be valid with an appropriate constant (e.g., C = −K log K) and a proper derivation, but the proof as written is not correct. The authors should provide a complete derivation that explicitly handles the factor K and the constant C, or state the corrected bound.","section":"Appendix A.3, Lemma 3"},{"comment":"The model assumes a single scalar correlation ρ shared by all pairs of experts and all latent dimensions, with off-diagonal entries σ_{i,j} = ρ σ_i σ_j in each Σ_d. This is a strong structural assumption about the error process of the encoders. The paper provides no diagnostic evidence that this assumption is adequate for the datasets considered; the sensitivity analyses in Table 12 and Fig. 6d vary only a global ρ, and no per-dimension or per-pair validation is reported. If the true error dependence is heterogeneous across dimensions or expert pairs, the CoDE posterior is misspecified and the reported improvements may not generalize. Please add an empirical check of the error covariance structure, or extend the parameterization to allow per-dimension or per-pair correlations, and discuss the effect of misspecification.","section":"Section 3.1, Lemma 1"}],"minor_comments":[{"comment":"There is a typo 'unonditional' in the title of the fourth panel of Fig. 4.","section":"Section 4.2 and Fig. 4"},{"comment":"There is a typo: 'On the otter hand' should be 'On the other hand'.","section":"Appendix D.5"},{"comment":"The sentence about the authors' mmJSD results disagreeing with (Huang et al., 2022) references a paper on GAN-based image synthesis, which appears unrelated to mmJSD; the citation is likely incorrect and the comment should be removed or rephrased.","section":"Appendix D.4"},{"comment":"The entropy loss is scaled by a factor of 1,000, said to be 'found by cross-validation,' but no details are given about the validation procedure; since this scale directly influences the learned π and the overall objective, please specify how it was selected.","section":"Appendix D.1"},{"comment":"The citation to (Cormen et al., 2022) for the elementary fact E[ξ|Xk] = π_k is unnecessary and would be clearer as a direct statement.","section":"Appendix A.3"},{"comment":"The model name is inconsistently capitalized as 'CoDe-VAE' in one place; please use 'CoDE-VAE' consistently.","section":"Appendix E.1"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection of ρ and the PolyMNIST FID-based architecture selection are the most serious issues and are likely to be raised by other reviewers. I recommend requiring a re-evaluation with a proper held-out validation protocol before acceptance. The gap in the proof of Lemma 3 may be fixable with a corrected derivation, but as written it is a mathematical error in a central result. Given the code release and the correctness of Lemma 2, the paper is worth a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two things well. First, it extends Winkler's consensus-of-experts formalism to multivariate latent spaces with correlated expert errors, giving the CoDE posterior in Lemma 2. That is a standard conjugate Gaussian calculation and it is correct under the stated Gaussian-error assumption. Second, it derives a weighted ELBO (Lemma 3) that learns subset contributions, which is a valid lower bound once the constant C is included, and it subsumes PoE as a special case. The authors also release code and run experiments on the standard benchmarks in the subfield. That is real, reproducible work and the methodological contribution is worth having even if the empirical claims turned out to be weaker than advertised.\n\nThe soft spots are concentrated in the evaluation protocol, not the math. The paper states that all experiments use the ρ value with the strongest overall performance, where ρ is exclusive to CoDE-VAE. That is test-set selection of a method-specific free parameter. Baselines do not get an equivalent selection step, so the comparison is not controlled. The PolyMNIST results are worse: 120 configurations are trained and the architecture with the smallest FID is chosen, again on the test set. With that protocol, the reported improvements in coherence/quality trade-off and log-likelihood cannot be cleanly attributed to modeling dependence. The ablation in Fig. 6 is subject to the same issue. The shared scalar ρ across all dimensions and expert pairs is restrictive, but that is a secondary concern; the evaluation protocol is the primary threat to the paper's central claim.\n\nOne more concrete issue: the Appendix B proof of positive definiteness is incorrect as written. The inequality step does not imply what the authors claim. The claim may still be true for their specific construction, but the proof needs fixing. The limitation section is honest about ρ's cross-validation cost, which I appreciate.\n\nWho is this for? Researchers working on multimodal VAEs who want a principled alternative to PoE/MoE aggregation and a starting point for handling dependent experts. The mathematical core and the released code make it worth engaging with seriously despite the evaluation flaws.\n\nMy recommendation: send it to peer review. The method is non-trivial, the derivations are mostly sound, and the code exists. But require a held-out validation protocol for ρ and a corrected appendix proof before accepting. I would not trust the current numbers as evidence of the headline claim.","headline":"A clean consensus-of-experts extension with a sound math core, but the headline empirical gains rest on test-set selection of ρ and the PolyMNIST architecture, so the paper needs a held-out validation protocol before the central claim is accepted.","tokens_in":27705,"tokens_out":1507,"would_cite":true,"duration_ms":18245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating single-modality encoders as dependent experts, not independent ones, improves multimodal VAEs on coherence, log-likelihood, and the generative quality gap.","keywords":["multimodal variational autoencoders","consensus of experts","dependent experts","evidence lower bound","generative coherence","generative quality","product of experts","mixture of experts"],"falsifier":"On a synthetic multimodal dataset with known ground-truth latent $z$, record the expert point-estimate errors $e_j = \\mu_j - z$ over many samples and latent dimensions; if the joint distribution of $(e_1, \\ldots, e_M)$ is not multivariate Gaussian, or the pairwise correlation varies materially across expert pairs or dimensions, then the fixed-$\\rho$ Gaussian assumption of Lemma 1 is violated and the CoDE posterior is misspecified, so its likelihood and coherence advantages over product-of-experts should weaken or disappear.","tokens_in":26758,"feed_emoji":"🧠","tokens_out":6201,"duration_ms":50663,"temperature":0.7,"pith_summary":"Multimodal variational autoencoders (VAEs) combine estimates from several single-modality 'expert' encoders into a joint latent posterior, yet the standard product- and mixture-of-experts rules assume the experts are independent. This paper argues that independence is overoptimistic because the modalities observe the same underlying object, and introduces consensus of dependent experts (CoDE), which models the correlation between experts' estimation errors. From CoDE it derives CoDE-VAE, a multimodal VAE whose objective learns how much each subset of modalities should contribute to the evidence lower bound. The paper reports that CoDE-VAE balances generative coherence against generative quality better than existing methods, produces tighter log-likelihood estimates, and narrows the generative quality gap as the number of modalities grows, matching unimodal VAEs in some settings.","feed_headline":"Treating modality experts as dependent boosts multimodal VAEs","feed_subtitle":"CoDE-VAE models correlations between single-modality encoders, improving coherence and log-likelihood.","key_machinery":"The load-bearing object is the CoDE posterior (Lemma 2): with expert point estimates $\\mu_k$ and their error covariances $\\Sigma_k$ arranged through a design matrix $u$, the consensus distribution for a subset is $h(\\theta_k|\\mu_k) \\sim \\mathcal{N}(A_k^{-1}B_k, A_k^{-1})$, where $A_k = u^T\\Sigma_k^{-1}u$ and $B_k = u^T\\Sigma_k^{-1}\\mu_k$. Dependence between experts is encoded in the off-diagonal entries $\\sigma_{i,j} = \\rho\\sigma_i\\sigma_j$ of each per-dimension error covariance $\\Sigma_d$, with a single correlation $\\rho$ chosen by cross-validation. The second piece is the ELBO of Eq. (3), which replaces equal subset weights with learnable categorical weights $\\pi_k$, so information-rich, more certain subsets contribute more to optimization.","core_discovery":"The paper's central claim is that a Bayesian posterior formed from expert point estimates, with a covariance matrix whose off-diagonal entries $\\rho\\sigma_i\\sigma_j$ encode the dependence between expert errors, yields a correct consensus distribution for multimodal VAEs: $h(\\theta_k|\\mu_k) \\sim \\mathcal{N}(A_k^{-1}B_k, A_k^{-1})$ (Lemma 2). This posterior subsumes the product of experts when $\\rho = 0$, leans toward more certain experts when $\\rho > 0$, and avoids both the variance underestimation of product-of-experts and the vagueness of mixture-of-experts. Used inside a new ELBO that learns weights $\\pi_k$ for each subset of modalities (Eq. 3), the method, CoDE-VAE, is shown empirically to improve the trade-off between generative coherence and generative quality, yield tighter log-likelihood estimates, and reduce the generative quality gap as more modalities are added, matching unimodal VAE quality in some cases.","pith_inferences":["Our inference: the single shared $\\rho$ could be replaced by per-pair or per-dimension correlations (or learned), which might further improve the consensus posterior on heterogeneous modalities at the cost of more parameters and heavier cross-validation.","Our inference: because CoDE only needs invertible $\\Sigma_d$, negative or dimension-specific correlations are allowed; testing anti-correlated experts (e.g., complementary views) is a natural extension the authors do not explore.","Our inference: the CoDE aggregation rule is not VAE-specific; it could be applied wherever expert distributions are combined, such as early-exit ensembles or LLM pairwise comparisons, as a drop-in replacement for product-of-experts.","Our inference: the learned $\\pi_k$ weights could double as a reliability signal for missing-modality inference, telling a deployed model which subset of available modalities to trust most."],"forward_implications":["CoDE-VAE reduces the generative quality gap as the number of modalities grows; with 4–5 PolyMNIST modalities its unconditional FID matches a unimodal VAE.","Because it avoids modality sub-sampling, CoDE-VAE yields tighter log-likelihood estimates than mixture-of-experts methods such as mmJSD and MMVAE.","The learned subset weights $\\pi_k$ correlate with certainty: subsets with lower covariance trace receive larger weights, so the objective automatically up-weights reliable modality combinations.","The CoDE posterior reduces to the product of experts when $\\rho = 0$, so CoDE-VAE is a strict generalization of PoE-based multimodal VAEs.","CoDE-VAE achieves classification accuracy comparable to leading multimodal VAEs, so modeling dependence does not hurt discriminative performance."],"supporting_citations":[{"why":"Supplies the consensus-of-experts principle (combining distributions from dependent information sources) that CoDE extends to multivariate data and VAEs.","marker":"(Winkler, 1981)"},{"why":"Introduces product of experts, the aggregation CoDE generalizes when $\\rho=0$ and whose variance underestimation motivates modeling dependence.","marker":"(Hinton, 2002)"},{"why":"Introduces MVAE, the PoE-based multimodal VAE baseline CoDE-VAE compares against and whose ELBO and aggregation CoDE modifies.","marker":"(Wu & Goodman, 2018)"},{"why":"Provides the generalized multimodal ELBO over all subsets $X_k$ (Eq. 1) which CoDE-VAE re-weights with learnable $\\pi_k$.","marker":"(Sutter et al., 2021)"},{"why":"Documents the generative quality gap and the harm of modality sub-sampling, the failure mode CoDE-VAE's no-sub-sampling design targets.","marker":"(Daunhawer et al., 2022)"},{"why":"Introduces MMVAE, the mixture-of-experts baseline that CoDE-VAE is compared against.","marker":"(Shi et al., 2019)"},{"why":"Introduces the MNIST-SVHN-Text and PolyMNIST datasets and the mmJSD baseline used in evaluation.","marker":"(Sutter et al., 2020)"},{"why":"Introduces MMVAE+, the strongest generative-quality baseline with modality-specific latents, and the CUB caption-coherence evaluation approach.","marker":"(Palumbo et al., 2023)"}],"fun_headline_variants":["Dependent expert consensus sharpens multimodal VAEs","CoDE-VAE: Correlating expert errors improves multimodal learning","Expert dependence tightens multimodal VAE bounds","Dependent experts close multimodal generative gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the joint error of all experts on every latent dimension is Gaussian with one shared correlation coefficient $\\rho$ for every pair of experts and every dimension; if expert errors are not Gaussian, or their dependence is not captured by a single scalar, the CoDE posterior is misspecified and the reported gains may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Dependent expert consensus sharpens multimodal VAEs","CoDE-VAE: Correlating expert errors improves multimodal learning","Expert dependence tightens multimodal VAE bounds","Dependent experts close multimodal generative gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2342,"prompt_tokens":949,"completion_tokens":1393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1342}},"tokens_in":565,"tokens_out":1393,"duration_ms":9231,"temperature":1.0,"reasoning_tokens":1342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:26:48.161747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic multimodal dataset with known ground-truth latent $z$, record the expert point-estimate errors $e_j = \\mu_j - z$ over many samples and latent dimensions; if the joint distribution of $(e_1, \\ldots, e_M)$ is not multivariate Gaussian, or the pairwise correlation varies materially across expert pairs or dimensions, then the fixed-$\\rho$ Gaussian assumption of Lemma 1 is violated and the CoDE posterior is misspecified, so its likelihood and coherence advantages over product-of-experts should weaken or disappear.","supporting_citations":[{"cited_title":"We assume Laplace likelihoods","cited_arxiv_id":null,"evidence_quote":"Documents the generative quality gap and the harm of modality sub-sampling, the failure mode CoDE-VAE's no-sub-sampling design targets."}],"review_version":1}