Pith. sign in

REVIEW 4 major objections 6 minor 42 references

Multimodal Variational Autoencoder: a Barycentric View

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that product-of-experts and mixture-of-experts aggregation in multimodal VAEs are both instances of the same barycenter problem—minimizing a weighted KL divergence in opposite directions—and introduces the Wasserstein…

desk verdict A clean barycentric reframing of PoE/MoE, plus a Wasserstein aggregation that works empirically but lacks the theoretical grounding the paper claims. read the letter →

arxiv 2412.20487 v1 pith:ZBIVBQNR submitted 2024-12-29 cs.LG cs.CVcs.ITmath.IT

Peijie Qiu , Wenhui Zhu , Sayantan Kumar , Xiwen Chen , Xiaotong Sun , Jin Yang , Abolfazl Razi , Yalin Wang
show 1 more author
Aristeidis Sotiras
This is my paper · ORCID
classification cs.LGcs.CVcs.ITmath.IT
keywords multimodalVAEbarycenterWassersteinproductofexpertsmixtureKLdivergencemissingmodalitieslatentrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal VAEs must combine a separate approximate posterior per modality into one joint posterior. This paper argues that the standard combination rules are not heuristic expert mixtures but are exactly the barycenters of the unimodal posteriors: product-of-experts minimizes a weighted reverse KL divergence and mixture-of-experts minimizes a weighted forward KL divergence. Seeing them this way exposes why PoE is zero-forcing and MoE is mass-covering, and opens the door to other divergences. The paper proposes the Wasserstein barycenter with equal weights, which for Gaussian unimodal posteriors is a Gaussian whose mean is the weighted mean of the means and whose covariance solves a fixed-point equation (and reduces to elementwise formulas for diagonal covariances). Experiments on PolyMNIST, MNIST-SVHN-TEXT, and CelebA indicate that this Wasserstein aggregation matches or improves latent representation quality and conditional generation coherence, particularly as the number of modalities grows.

What carries the argument

The machinery is the barycenter problem: for distributions $\{P_m\}$ and weights $\{\lambda_m\}$, a barycenter is $P_B = \arg\min_P \sum_m \lambda_m\, d(P_m, P)$ for a chosen divergence $d$. Choosing $d$ as reverse KL reproduces PoE, choosing it as forward KL reproduces MoE, and choosing it as the squared 2-Wasserstein distance produces the Wasserstein barycenter. In the Gaussian case the Wasserstein barycenter is the Bures-Wasserstein barycenter, whose covariance solves the fixed-point equation $\tilde{\Sigma} = \sum_m \lambda_m (\tilde{\Sigma}^{1/2} \Sigma_m \tilde{\Sigma}^{1/2})^{1/2}$, and in the isotropic diagonal case it reduces to the elementwise formulas $\tilde{\mu} = \sum_m \lambda_m \mu_m$ and $\tilde{\sigma} = \sum_m \lambda_m \sigma_m$. The barycenter computation is decoupled from the ELBO training, which is what makes the divergence choice a design decision rather than a fixed expert rule.

What would settle it

In a two-modality synthetic example where the true posterior is computable—say a shared latent $z$ observed through Gaussian likelihoods with very different variances—compare the equal-weight Wasserstein barycenter of the two unimodal posteriors with the true posterior in squared 2-Wasserstein distance. If the distance stays small across a wide range of variance imbalances, the equal-weight prescription is validated; if it grows sharply when one modality is much more informative than the other, the fixed weights are the weak point and a learned weighting would be needed.

Watch

Extended reading notes

Core claim

The paper establishes a unification result: the two dominant aggregation rules in multimodal variational autoencoders are barycenters under the two directions of KL divergence. Minimizing $\sum_m \lambda_m D_{\mathrm{KL}}(q \,\|\, q_{\phi_m})$ over $q$, with weights $\lambda_m$ summing to one, yields the product-of-experts posterior (up to normalization), while minimizing $\sum_m \lambda_m D_{\mathrm{KL}}(q_{\phi_m} \,\|\, q)$ yields the mixture-of-experts posterior. This is the content of Theorem 1, with the normalization subtlety of PoE handled in Appendix A.2. The paper then replaces KL with the squared 2-Wasserstein distance and defines WB-VAE, whose approximate joint posterior is the Wasserstein barycenter of the unimodal Gaussian posteriors. For isotropic Gaussian encoders this barycenter is explicit: the joint mean is $\tilde{\mu} = \sum_m \lambda_m \mu_m$ and the joint scale is $\tilde{\sigma} = \sum_m \lambda_m \sigma_m$ in each dimension, interpolating between modalities along the Wasserstein geodesic rather than forcing or erasing support.

Load-bearing premise

The load-bearing assumption is that the true joint posterior is well approximated by the equal-weight Wasserstein barycenter of the per-modality Gaussian posteriors, which the paper motivates by geometric intuition about geodesic interpolation but does not support with a closeness guarantee or learned weights.

Editorial extensions

If this is right

  • PoE and MoE are not competing heuristics but two sides of KL asymmetry: PoE minimizes reverse KL and is therefore zero-forcing, while MoE minimizes forward KL and is therefore mass-covering.
  • Any divergence that is convex in the unimodal distributions yields a valid ELBO and scalable inference through Jensen's inequality, opening the same barycentric framework to $f$-divergences, Gromov-Wasserstein distances, or other metric divergences.
  • Because the 2-Wasserstein distance is symmetric, the WB-VAE posterior is neither zero-forcing nor mass-covering; it sits between them, and on PolyMNIST its metrics improve roughly linearly as modalities are added, unlike the saturating behavior of the KL-based baselines.
  • The mixture-of-barycenters variant MWB-VAE outperforms MoPoE-VAE on most evaluated tasks in the paper, including the difficult image-to-text and text-to-image generation directions on CelebA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not prove that the equal-weight Wasserstein barycenter approximates the true joint posterior; a formal bound on $W_2(\tilde{q}_{\mathrm{WB}}, p_{\theta}(z \mid X_{1:M}))$, or a counterexample with strongly heteroscedastic modalities, would decide how far the prescription generalizes.
  • Since the barycentric view separates aggregation from training, the fixed weights $\lambda_m = 1/M$ could be replaced by learned or uncertainty-scaled weights without changing the framework; that is a direct extension the paper leaves implicit.
  • The same barycentric lens could be applied to other generative latent variable models beyond VAEs, such as multimodal diffusion models, where an aggregation of per-modality reverse processes could be chosen by a metric divergence rather than by a product rule.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a barycentric reformulation of aggregation functions in multimodal VAEs. It argues that product-of-experts (PoE) and mixture-of-experts (MoE) aggregation are specific instances of barycenters obtained by minimizing reverse and forward KL divergences, respectively (Theorem 1). It then introduces WB-VAE, which aggregates unimodal Gaussian posteriors via the 2-Wasserstein barycenter, with a closed-form isotropic-Gaussian solution (Remark 2), and a mixture variant MWB-VAE (Remark 3). The method is evaluated on PolyMNIST, MNIST-SVHN-TEXT, and CelebA, reporting competitive classification accuracy, generation coherence, and log-likelihood.

Significance. The barycentric interpretation of PoE and MoE in Theorem 1 is a genuine and clearly stated theoretical contribution: the derivation in Appendix A.2 correctly identifies PoE as the weighted reverse-KL barycenter and MoE as the weighted forward-KL barycenter, and it provides an information-theoretic vocabulary (zero-forcing versus mass-covering) that is useful for reasoning about existing methods. However, the central methodological claim of the paper, namely that the equal-weight Wasserstein barycenter of unimodal posteriors is a sound approximation to the true joint posterior, is not supported and is in fact contradicted by the basic statistical property that a joint posterior under conditional independence should be sharper than every unimodal posterior. The proposed WB-VAE's aggregation rule, Eq. (7), averages standard deviations and therefore cannot produce a distribution sharper than the sharpest expert. This is a structural flaw in the proposed method, not a tuning issue. The empirical results, while sometimes favorable on classification and coherence metrics, do not test the sharpening property and cannot compensate for the lack of a theoretical justification.

major comments (4)
  1. [Multimodal VAE from Wasserstein Barycenter, Remark 2 and Eq. (7)] The equal-weight isotropic Wasserstein barycenter cannot fuse evidence. In Remark 2, Eq. (7) gives \tilde{\sigma} = (1/M)\sum_m \sigma_m, so the aggregate standard deviation is an arithmetic mean of the unimodal standard deviations. In contrast, for conditionally independent modalities, the true joint posterior has precision 1/\sigma_{\text{true}}^2 = 1/\sigma_{\text{prior}}^2 + \sum_m 1/\sigma_m^2, implying \sigma_{\text{true}} < \min_m \sigma_m. Thus the proposed WB-VAE posterior is never sharper than the sharpest unimodal posterior, and it dilutes informative modalities when uninformative ones are present. This is a structural property of the chosen divergence and weights, not a tuning issue. The paper's claim that the Wasserstein barycenter 'strikes a balance' or 'preserves geometry' does not address this failure to meet the defining requirement of a joint posterior in a multimodal VAE.
  2. [Proposition 1 and Appendix A.1] The claim that 'for any divergence measure d(q_{\phi_m}, \cdot) that is convex on q_{\phi_m}, the resultant barycenter guarantees a valid ELBO' is either trivial or unproved. The ELBO inequality log p(X_{1:M}) \geq L(\theta,\phi; X_{1:M}) holds for any approximate posterior q(z|X_{1:M}) whatsoever, regardless of how q is constructed, because it is equivalent to the nonnegativity of D_{KL}(q||p_{\theta}(z|X_{1:M})). The proof in Appendix A.1 indeed only notes that D_{KL}(f(\mathcal{M}(...))||p) \geq 0, which is always true and does not use the barycenter minimization or Jensen's inequality in any substantive way. The additional statement that the left-hand side of Eq. (5) 'defines a scalable inference' while the right-hand side requires 2^M inference networks is also unsupported: the mixture \sum_m \lambda_m q_{\phi_m} is not generally the minimizer of the right-hand side except in the special forward-KL case. Proposition 1 therefore provides no connection between the barycenter construction and the quality of the approximation to the true joint posterior.
  3. [Multimodal VAE from Wasserstein Barycenter, paragraph after Eq. (7)] The paper chooses the barycenter weights \lambda_m = 1/M without any theoretical or empirical justification, stating only that 'it is typically safe' to do so. No learned-weight scheme, no error bound, and no condition are given under which the equal-weight Wasserstein barycenter is close to the true joint posterior. This is a load-bearing omission because the weights are the only free parameter of the proposed aggregation function: with equal weights, the method reduces to a simple average of means and standard deviations, which is not a principled approximation of the joint posterior. The claim that different weights 'may lead to a joint posterior that maintains diverse shapes and structures' is speculative and not tested. A concrete test would be to measure the entropy or precision of \tilde{q} as the number of modalities increases and compare it to the true conditional-independent posterior; such a test is absent.
  4. [Experiments, Tables 1-4 and Figure 3] The empirical evaluation does not resolve the structural concern above. The reported log-likelihoods (Table 3) show WB-VAE is on par with or worse than PoE-VAE, and the classification/coherence gains of WB-VAE and MWB-VAE can be explained by the smoothing effect of averaging rather than by successful joint-posterior approximation. The claim that WB-VAE 'strikes a balance' between zero-forcing and mass-covering is a restatement of the averaging behavior, not evidence that the aggregate posterior sharpens with additional modalities. The paper needs a direct evaluation of the sharpening property, for instance by computing the precision or differential entropy of \tilde{q}(z|x_{1:M}) as a function of M, or by comparing against the analytical joint posterior under conditional independence. Without such evidence, the central motivation for WB-VAE remains unsupported.
minor comments (6)
  1. [Abstract] The phrase 'to for multimodal representation learning' contains a typo; it should read 'for multimodal representation learning'.
  2. [Method, Eq. (3)] The reconstruction term E_{q}[\log p_{\theta}(X_{1:M}|z)] is written with a single joint likelihood; for conditionally independent modalities it should factorize as \sum_m E_{q}[\log p_{\theta_m}(x_m|z)]. As written, the connection between the ELBO and the unimodal decoders is ambiguous.
  3. [Proposition 1 and Eq. (5)] The statement 'convex on q_{\phi_m}' is imprecise. The Jensen inequality in Eq. (5) requires d(\cdot, q) to be convex in its first argument for fixed q, not merely 'convex on' the unimodal distributions. Please clarify the convexity assumption and state the argument ordering consistently.
  4. [Appendix A.3] The proof of Remark 2 correctly derives \tilde{\mu} = \sum_m \lambda_m \mu_m and \tilde{\sigma} = \sum_m \lambda_m \sigma_m by differentiating the 1D squared 2-Wasserstein distance. However, the sentence 'the same results can also be derived by leveraging Proposition 1' is misleading, because Proposition 1's Jensen inequality does not identify the minimizer; remove or rephrase it.
  5. [Experiments and Supplementary Material] There are several typos: 'Folliwing' should be 'Following' in the MNIST-SVHN-TEXT dataset description; 'pf 32' should be 'of 32' in Section B.3; and 'V AE' appears with inconsistent spacing throughout. The caption of Figure 2 also contains an empty placeholder '( )' after 'Wasserstein barycenter'.
  6. [Related Work] The paper cites Sutter, Daunhawer, and Vogt (2021) for the generalized multimodal ELBO but does not discuss how the proposed barycentric formulation relates to the sub-sampled ELBO objective or to the mixture-of-product-of-experts construction beyond the brief remark in Appendix A.4. A more detailed comparison would help position the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the PoE/MoE-as-barycenter results are mathematical equivalences, and the WB-VAE aggregation is derived from standard optimal transport with an explicit, non-fitted equal-weight convention.

full rationale

The paper's central derivation is self-contained. Theorem 1 proves that PoE is the minimizer of the weighted reverse KL divergence and MoE is the minimizer of the weighted forward KL divergence; this is an equivalence proof from the definitions of the barycenter problem and the KL divergence, not an assumption of the conclusion. The Bures-Wasserstein barycenter formula in Eq. (6) and its isotropic simplification in Eq. (7) follow from cited analytical results for Gaussian optimal transport (Agueh and Carlier; Knott and Smith), which are external and not authored by the present paper. The weight choice lambda_m = 1/M is explicit ('it is typically safe to set lambda_m = 1/M, forall m') and is not fitted to the benchmark results, so it does not make the empirical outcomes forced. The empirical evaluation is against external baselines on standard benchmarks with shared architectures. The paper's weaker point, that the equal-weight Wasserstein barycenter approximates the true joint posterior, is an unsupported modeling assumption rather than a circular step: no equation in the paper defines the true posterior in terms of the barycenter, and no fitted parameter is relabeled as a prediction. Therefore no load-bearing step reduces to its own inputs, and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests primarily on the standard VAE Gaussian-posterior assumption and the convention lambda=1/M. No new entities are introduced. The main unverified assumption is the convexity requirement in Proposition 1 and the adequacy of the W2 barycenter as an approximation to the joint posterior.

free parameters (2)
  • barycenter weights lambda_m = 1/M for all m (equal weights)
    The barycentric framework allows arbitrary weights, but the paper fixes them to 1/M without learning or justification beyond convention. The choice affects the joint posterior and therefore the ELBO, but it is not fitted to data.
  • KL weight balance beta = 2.5 (PolyMNIST, CelebA), 5.0 (MNIST-SVHN-TEXT)
    Standard VAE beta hyperparameter from the experimental protocol of Sutter et al. 2021; not central to the barycentric claim but part of the empirical setup.
assumptions (3)
  • domain assumption Unimodal inference distributions q_phi_m(z|x_m) are Gaussian with diagonal covariance.
    This is the standard VAE assumption (Kingma & Welling 2013) and is stated in the Method section before Remark 2. The closed-form Bures-Wasserstein barycenter and the simple averaging formula in Eq. (7) depend on it.
  • ad hoc to paper The 2-Wasserstein-squared distance is convex in the linear-mixture sense required by Proposition 1's Jensen inequality.
    Proposition 1 asserts the Jensen bound d(sum lambda_m q_m, q) <= sum lambda_m d(q_m, q) for 'any divergence measure d convex on q_m'. The paper does not verify that W2^2 satisfies this in the linear mixture sense; in fact, W2^2 is displacement-convex, not generally convex in the linear sense. The proposition is not used to derive WB-VAE, but it is stated as a general guarantee.
  • domain assumption A bilevel optimization where the lower level (barycenter) and upper level (ELBO) are solved separately and both optimally is a valid description of multimodal VAE training.
    Lemma 1 describes the training as bilevel, but in practice the barycenter is computed in closed form and the encoders are trained end-to-end. The paper does not analyze the interaction between the two levels, e.g., gradient flow through the barycenter parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Variational Autoencoder: a Barycentric View." pith.science (2026). https://pith.science/paper/ZBIVBQNR

@misc{pith2026241220487,
  author       = {Pith},
  title        = {Pith review of: Multimodal Variational Autoencoder: a Barycentric View},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBIVBQNR}},
  note         = {Machine review of arXiv:2412.20487}
}
read the original abstract

Multiple signal modalities, such as vision and sounds, are naturally present in real-world phenomena. Recently, there has been growing interest in learning generative models, in particular variational autoencoder (VAE), to for multimodal representation learning especially in the case of missing modalities. The primary goal of these models is to learn a modality-invariant and modality-specific representation that characterizes information across multiple modalities. Previous attempts at multimodal VAEs approach this mainly through the lens of experts, aggregating unimodal inference distributions with a product of experts (PoE), a mixture of experts (MoE), or a combination of both. In this paper, we provide an alternative generic and theoretical formulation of multimodal VAE through the lens of barycenter. We first show that PoE and MoE are specific instances of barycenters, derived by minimizing the asymmetric weighted KL divergence to unimodal inference distributions. Our novel formulation extends these two barycenters to a more flexible choice by considering different types of divergences. In particular, we explore the Wasserstein barycenter defined by the 2-Wasserstein distance, which better preserves the geometry of unimodal distributions by capturing both modality-specific and modality-invariant representations compared to KL divergence. Empirical studies on three multimodal benchmarks demonstrated the effectiveness of the proposed method.

Figures

Figures reproduced from arXiv: 2412.20487 by the authors.

Figure 1
Figure 1. The overview of a multimodal VAE that takes [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of methods for aggregating the unimodal inference distributions ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Quantitative results on PolyMNIST as a function of the number of input modalities, averaged over all subsets of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Conditionally generated images given the text on top of each column on bimodal CelebA using [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Agueh, M.; and Carlier, G. 2011. Barycenters in the W asserstein space. SIAM Journal on Mathematical Analysis, 43(2): 904--924

    work page 2011

  4. [4]

    Ambrosio, L.; Gigli, N.; and Savar \'e , G. 2008. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media

    work page 2008

  5. [5]

    Arjovsky, M.; Chintala, S.; and Bottou, L. 2017. Wasserstein GAN. arXiv:1701.07875

    arXiv 2017

  6. [6]

    Baltru s aitis, T.; Ahuja, C.; and Morency, L.-P. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2): 423--443

    2018

  7. [7]

    Cover, T. M. 1999. Elements of information theory. John Wiley & Sons

    1999

  8. [8]

    R.; and Shortt, R

    Givens, C. R.; and Shortt, R. M. 1984. A class of Wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31(2): 231--240

    work page 1984

Show all 42 references
  1. [9]

    Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. Advances in neural information processing systems, 27

    2014

  2. [10]

    Grove, K.; and Karcher, H. 1973. How to conjugateC1-close group actions. Mathematische Zeitschrift, 132: 11--20

    1973

  3. [11]

    Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; and Courville, A. C. 2017. Improved training of wasserstein gans. Advances in neural information processing systems, 30

    2017

  4. [12]

    P.; Bosnjak, M.; Shanahan, M.; Botvinick, M.; Hassabis, D.; and Lerchner, A

    Higgins, I.; Sonnerat, N.; Matthey, L.; Pal, A.; Burgess, C. P.; Bosnjak, M.; Shanahan, M.; Botvinick, M.; Hassabis, D.; and Lerchner, A. 2017. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv:1707.03389

    2017 arXiv

  5. [13]

    Hirt, M.; Campolo, D.; Leong, V.; and Ortega, J.-P. 2024. Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives. Transactions on Machine Learning Research

    2024

  6. [14]

    Kantorovich, L. V. 1942. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), volume 37, 199--201

    1942

  7. [15]

    Karpathy, A.; and Fei-Fei, L. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3128--3137

    2015

  8. [16]

    Kingma, D. P. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  9. [17]

    P.; and Welling, M

    Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  10. [18]

    Knott, M.; and Smith, C. S. 1984. On the optimal mapping of distributions. Journal of Optimization Theory and Applications, 43: 39--49

    1984

  11. [19]

    Knott, M.; and Smith, C. S. 1994. On a generalization of cyclic monotonicity and distances among random vectors. Linear algebra and its applications, 199: 363--371

    1994

  12. [20]

    Korthals, T.; Rudolph, D.; Leitner, J.; Hesse, M.; and R \"u ckert, U. 2019. Multi-modal generative models for learning epistemic active sensing. In 2019 International Conference on Robotics and Automation (ICRA), 3319--3325. IEEE

    2019

  13. [21]

    LeCun, Y.; and Cortes, C. 2010. MNIST handwritten digit database

    2010

  14. [22]

    Lin, Y.-B.; Sung, Y.-L.; Lei, J.; Bansal, M.; and Bertasius, G. 2023. Vision transformers are parameter-efficient audio-visual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2299--2309

    2023

  15. [23]

    Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, 3730--3738

    2015

  16. [24]

    Minka, T.; et al. 2005. Divergence measures and message passing. Technical report, Technical report, Microsoft Research

    2005

  17. [25]

    Miyato, T.; Kataoka, T.; Koyama, M.; and Yoshida, Y. 2018. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957

    2018 arXiv

  18. [26]

    Monge, G. 1781. M \'e moire sur la th \'e orie des d \'e blais et des remblais. Mem. Math. Phys. Acad. Royale Sci., 666--704

  19. [27]

    Murphy, K. P. 2012. Machine learning: a probabilistic perspective. MIT press

    2012

  20. [28]

    Y.; et al

    Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; Ng, A. Y.; et al. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, 4. Granada

    2011

  21. [29]

    Ngiam, J.; Khosla, A.; Kim, M.; Nam, J.; Lee, H.; and Ng, A. Y. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), 689--696

    2011

  22. [30]

    Palumbo, E.; Daunhawer, I.; and Vogt, J. E. 2023. MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises. In The Eleventh International Conference on Learning Representations. OpenReview

    2023

  23. [31]

    Peyr \'e , G.; Cuturi, M.; et al. 2019. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11(5-6): 355--607

    2019

  24. [32]

    P.; Manzini, T.; Morency, L.-P.; and P \'o czos, B

    Pham, H.; Liang, P. P.; Manzini, T.; Morency, L.-P.; and P \'o czos, B. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 6892--6899

    2019

  25. [33]

    Schonfeld, E.; Ebrahimi, S.; Sinha, S.; Darrell, T.; and Akata, Z. 2019. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8247--8255

    2019

  26. [34]

    Shi, Y.; Paige, B.; Torr, P.; et al. 2019. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in neural information processing systems, 32

    2019

  27. [35]

    Sutter, T.; Daunhawer, I.; and Vogt, J. 2020. Multimodal generative learning utilizing jensen-shannon-divergence. Advances in neural information processing systems, 33: 6100--6110

    2020

  28. [36]

    M.; Daunhawer, I.; and Vogt, J

    Sutter, T. M.; Daunhawer, I.; and Vogt, J. E. 2021. Generalized multimodal ELBO. arXiv preprint arXiv:2105.02470

    2021 arXiv

  29. [37]

    Suzuki, M.; and Matsuo, Y. 2022. A survey of multimodal deep generative models. Advanced Robotics, 36(5-6): 261--278

    2022

  30. [38]

    Suzuki, M.; Nakayama, K.; and Matsuo, Y. 2016. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891

    2016 arXiv

  31. [39]

    Turner, R.; and Sahani, M. 2011. Two problems with variational expectation maximisation for time-series models. Cambridge University Press

    2011

  32. [40]

    Vedantam, R.; Fischer, I.; Huang, J.; and Murphy, K. 2017. Generative models of visually grounded imagination. arXiv preprint arXiv:1705.10762

    2017 arXiv

  33. [41]

    Wu, M.; and Goodman, N. 2018. Multimodal generative models for scalable weakly-supervised learning. Advances in neural information processing systems, 31

    2018

  34. [42]

    Yuan, S.; Cui, J.; Li, H.; and Han, T. 2024. Learning Multimodal Latent Generative Models with Energy-Based Prior. arXiv preprint arXiv:2409.19862

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.