pith. sign in

arxiv: 2606.11614 · v1 · pith:ZNKI5OHHnew · submitted 2026-06-10 · 💻 cs.LG · cs.AI· cs.CV

Information-Theoretic Decomposition for Multimodal Interaction Learning

Zequn Yang , Yake Wei , Haotian Ni , Zhihao Xu , Di Hu This is my paper

Pith reviewed 2026-06-27 10:23 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CV
keywords multimodal learninginformation-theoretic decompositionsample-specific interactionsredundant unique synergisticvariational architectureinteraction learningfine-tuning strategy
0
0 comments X

The pith

DMIL uses variational decomposition to isolate and learn from sample-specific redundant, unique, and synergistic multimodal interactions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that multimodal interactions vary dynamically across individual samples and that conventional approaches fall short because ensembles miss synergies while joint training underuses redundancies. An information-theoretic analysis shows why adapting to these sample-specific patterns matters for effective learning. The proposed method first applies a variational decomposition to separate the interaction components explicitly, then uses a fine-tuning strategy that leverages those components. Experiments across tasks and architectures indicate consistent performance gains from this per-sample adaptation. The result points toward an interaction-centric way of building multimodal models.

Core claim

By designing a variational decomposition architecture to isolate redundant, unique, and synergistic interaction components on a per-sample basis and then applying a learning strategy that incorporates these explicit components during fine-tuning, the approach enables comprehensive interaction learning that adapts holistically to each sample.

What carries the argument

The variational decomposition architecture that isolates redundant, unique, and synergistic multimodal interaction components on a per-sample basis.

If this is right

  • Modality ensemble methods fail to capture synergy while joint learning paradigms under-utilize redundant information.
  • Adapting to sample-specific interactions produces superior performance across diverse tasks and model architectures.
  • The framework applies flexibly to different multimodal setups without requiring architecture-specific changes.
  • An interaction-centric paradigm replaces task-specific heuristics with explicit decomposition and learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The per-sample decomposition could support post-hoc analysis of which interaction type drives a model's decision on any given input.
  • Similar decomposition ideas might transfer to non-multimodal settings where information sources interact dynamically, such as sensor fusion in robotics.
  • If the components prove stable across training runs, they could serve as regularizers in other multimodal training pipelines.

Load-bearing premise

The variational decomposition can reliably separate the three interaction types per sample without significant leakage or misattribution between components.

What would settle it

On a controlled synthetic dataset where the true redundant, unique, and synergistic information amounts are known in advance, check whether the decomposed components recover those known quantities with low error.

Figures

Figures reproduced from arXiv: 2606.11614 by Di Hu, Haotian Ni, Yake Wei, Zequn Yang, Zhihao Xu.

Figure 3
Figure 3. Figure 3: The DMIL framework incorporates two dis [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
read the original abstract

Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities, which collectively constitute multimodal interactions. A critical yet underexplored challenge is that these implicit interactions vary dynamically across samples. In this work, we present the first systematic, information-theoretic analysis highlighting why learning these dynamic, sample-specific interactions is critical for effective multimodal learning. Our analysis further reveals deficits in conventional paradigms at learning these distinct interaction types: modality ensemble approaches struggle to capture synergy, while joint learning paradigms often under-utilize redundant information. This highlights the need for an approach that can adaptively learn from different interaction types on a per-sample basis. To this end, we propose Decomposition-based Multimodal Interaction Learning (DMIL), a novel paradigm that explicitly models and learns from sample-specific interactions. First, we design a variational decomposition architecture to isolate the constituent interaction components. Second, we employ a new learning strategy that leverages these explicit interaction components in a fine-tuning process to achieve comprehensive interaction learning. Extensive experiments across diverse tasks and architectures demonstrate that DMIL consistently achieves superior performance by adapting to holistic sample-specific interactions. Our framework is flexible and broadly applicable, establishing an interaction-centric paradigm for multimodal learning. The code is available at https://github.com/GeWu-Lab/DMIL.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 0 minor

Summary. The paper claims that multimodal interactions (redundant, unique, synergistic) vary dynamically across samples, that conventional modality-ensemble and joint-learning paradigms are deficient at capturing them, and that the proposed DMIL framework—via a variational decomposition architecture that isolates per-sample components followed by a fine-tuning strategy—achieves superior performance across tasks and architectures by adapting to holistic sample-specific interactions. Code is released.

Significance. If the variational decomposition reliably isolates the three interaction types without leakage or misattribution, the work would supply an interaction-centric paradigm that directly addresses a documented limitation of existing multimodal methods; the public code release is a concrete strength for reproducibility.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their summary of our work and for noting the potential impact if the variational decomposition reliably isolates interaction types. We appreciate the recognition of the code release for reproducibility. No specific major comments were provided in the report, so we have no point-by-point revisions to address. We are happy to provide additional clarifications if requested.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The abstract and provided text contain no equations, objective functions, or derivation steps. No variational decomposition architecture, information-theoretic bounds, or learning strategy is formalized with math that could reduce to fitted inputs or self-citations. The reader's assessment correctly notes that circularity cannot be assessed without the full manuscript; absent any load-bearing claims that quote to self-referential definitions or predictions-by-construction, the score is 0. This is the expected honest non-finding when no technical chain is visible.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the core premise that multimodal interactions decompose into redundant/unique/synergistic components is treated as a domain assumption drawn from information theory.

axioms (1)
  • domain assumption Multimodal interactions can be decomposed into redundant, unique, and synergistic components that vary dynamically across samples.
    This decomposition is presented as the foundation for both the analysis and the DMIL architecture in the abstract.

pith-pipeline@v0.9.1-grok · 5763 in / 1249 out tokens · 26024 ms · 2026-06-27T10:23:25.503335+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 16 canonical work pages · 4 internal anchors

  1. [1]

    Food-101–mining discriminative components with random forests,

    L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101–mining discriminative components with random forests,” inComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13. Springer, 2014, pp. 446–461

    2014

  2. [2]

    Multimodal markers of irony and sarcasm,

    S. Attardo, J. Eisterhold, J. Hay, and I. Poggi, “Multimodal markers of irony and sarcasm,”Humor: International Journal of Humor Research, 2003

    2003

  3. [3]

    Quantifying interactions in semi-supervised multimodal learning: Guarantees and applications,

    P. P. Liang, C. K. Ling, Y . Cheng, A. Obolenskiy, Y . Liu, R. Pandey, A. Wilf, L.-P. Morency, and R. Salakhutdinov, “Quantifying interactions in semi-supervised multimodal learning: Guarantees and applications,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  4. [4]

    A novel approach for effective multi-view clustering with information-theoretic perspective,

    C. Cui, Y . Ren, J. Pu, J. Li, X. Pu, T. Wu, Y . Shi, and L. He, “A novel approach for effective multi-view clustering with information-theoretic perspective,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  5. [5]

    Cross-modal consistency in multimodal large language models,

    X. Zhang, S. Li, N. Shi, B. Hauer, Z. Wu, G. Kondrak, M. Abdul-Mageed, and L. V . Lakshmanan, “Cross-modal consistency in multimodal large language models,”arXiv preprint arXiv:2411.09273, 2024

    work page arXiv 2024

  6. [6]

    Factorized contrastive learning: Going beyond multi-view redundancy,

    P. P. Liang, Z. Deng, M. Q. Ma, J. Y . Zou, L.-P. Morency, and R. Salakhutdinov, “Factorized contrastive learning: Going beyond multi-view redundancy,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  7. [7]

    What to align in multimodal contrastive learning?

    B. Dufumier, J. Castillo-Navarro, D. Tuia, and J.-P. Thiran, “What to align in multimodal contrastive learning?”arXiv preprint arXiv:2409.07402, 2024

    work page arXiv 2024

  8. [8]

    Multimodal learning without labeled multimodal data: Guarantees and applications,

    P. P. Liang, C. K. Ling, Y . Cheng, A. Obolenskiy, Y . Liu, R. Pandey, A. Wilf, L.-P. Morency, and R. Salakhutdinov, “Multimodal learning without labeled multimodal data: Guarantees and applications,”arXiv preprint arXiv:2306.04539, 2023

    work page arXiv 2023

  9. [9]

    Efficient quantification of multimodal interaction at sample level,

    Z. Yang, H. Wang, and D. Hu, “Efficient quantification of multimodal interaction at sample level,” inForty-Second International Conference on Machine Learning, 2025

    2025

  10. [10]

    Multimodal fusion balancing through game-theoretic regularization,

    K. Kontras, T. Strypsteen, C. Chatzichristos, P. P. Liang, M. Blaschko, and M. De V os, “Multimodal fusion balancing through game-theoretic regularization,”arXiv preprint arXiv:2411.07335, 2024

    work page arXiv 2024

  11. [11]

    Quantifying and enhancing multi-modal robustness with modality preference,

    Z. Yang, Y . Wei, C. Liang, and D. Hu, “Quantifying and enhancing multi-modal robustness with modality preference,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  12. [12]

    Balanced multimodal learning via on-the-fly gradient modulation,

    X. Peng, Y . Wei, A. Deng, D. Wang, and D. Hu, “Balanced multimodal learning via on-the-fly gradient modulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8238–8247

    2022

  13. [13]

    Reconboost: Boosting can achieve modality reconcilement,

    C. Hua, Q. Xu, S. Bao, Z. Yang, and Q. Huang, “Reconboost: Boosting can achieve modality reconcilement,”arXiv preprint arXiv:2405.09321, 2024

    work page arXiv 2024

  14. [14]

    Deep Variational Information Bottleneck

    A. A. Alemi, I. Fischer, J. V . Dillon, and K. Murphy, “Deep variational information bottleneck,”arXiv preprint arXiv:1612.00410, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  15. [15]

    Flamingo: a Visual Language Model for Few-Shot Learning

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, A. Mensch, K. Millican, M. Reynolds, R. Ringet al., “Flamingo: a visual language model for few-shot learning,”arXiv preprint arXiv:2204.14198, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  16. [16]

    Gpt-4v(ision) technical report,

    OpenAI, “Gpt-4v(ision) technical report,” https://cdn.openai.com/papers/GPT-4V(ision).pdf, 2023

    2023

  17. [17]

    Crab: A unified audio-visual scene understanding model with explicit cooperation,

    H. Du, G. Li, C. Zhou, C. Zhang, A. Zhao, and D. Hu, “Crab: A unified audio-visual scene understanding model with explicit cooperation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 18 804–18 814

    2025

  18. [18]

    What makes multi-modal learning better than single (provably),

    Y . Huang, C. Du, Z. Xue, X. Chen, H. Zhao, and L. Huang, “What makes multi-modal learning better than single (provably),” Advances in Neural Information Processing Systems, vol. 34, pp. 10 944–10 956, 2021

    2021

  19. [19]

    A theory of multimodal learning,

    Z. Lu, “A theory of multimodal learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 57 244–57 255, 2023

    2023

  20. [20]

    On the computational benefit of multimodal learning,

    ——, “On the computational benefit of multimodal learning,” inInternational Conference on Algorithmic Learning Theory. PMLR, 2024, pp. 810–821

    2024

  21. [21]

    Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably),

    Y . Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, “Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably),”arXiv preprint arXiv:2203.12221, 2022. 11

    work page arXiv 2022

  22. [22]

    Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies,

    I. Gat, I. Schwartz, A. Schwing, and T. Hazan, “Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies,”Advances in Neural Information Processing Systems, vol. 33, pp. 3197–3208, 2020

    2020

  23. [23]

    Understanding unimodal bias in multimodal deep linear networks,

    Y . Zhang, P. E. Latham, and A. M. Saxe, “Understanding unimodal bias in multimodal deep linear networks,” inForty-first International Conference on Machine Learning, 2024

    2024

  24. [24]

    On the importance of contrastive loss in multimodal learning,

    Y . Ren and Y . Li, “On the importance of contrastive loss in multimodal learning,”arXiv preprint arXiv:2304.03717, 2023

    work page arXiv 2023

  25. [25]

    S i: Score-based o-information estimation,

    M. Bounoua, G. Franzese, and P. Michiardi, “S i: Score-based o-information estimation,” inICML 2024, 41st International Conference on Machine Learning, 2024

    2024

  26. [26]

    Learning robust representations via multi-view information bottleneck,

    M. Federici, A. Dutta, P. Forré, N. Kushman, and Z. Akata, “Learning robust representations via multi-view information bottleneck,”arXiv preprint arXiv:2002.07017, 2020

    work page arXiv 2002

  27. [27]

    Mibench: Evaluating lmms on multimodal interaction,

    Y . Miao, Z. Yang, Y . Wei, Z. Chen, H. Ni, H. Duan, K. Chen, and D. Hu, “Mibench: Evaluating lmms on multimodal interaction,” arXiv preprint arXiv:2603.13427, 2026

    work page arXiv 2026

  28. [28]

    Estimating the unique information of continuous variables,

    A. Pakman, A. Nejatbakhsh, D. Gilboa, A. Makkeh, L. Mazzucato, M. Wibral, and E. Schneidman, “Estimating the unique information of continuous variables,”Advances in neural information processing systems, vol. 34, pp. 20 295–20 307, 2021

    2021

  29. [29]

    Quantifying unique information,

    N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay, “Quantifying unique information,”Entropy, vol. 16, no. 4, pp. 2161–2183, 2014

    2014

  30. [30]

    Nonnegative Decomposition of Multivariate Information

    P. L. Williams and R. D. Beer, “Nonnegative decomposition of multivariate information,”arXiv preprint arXiv:1004.2515, 2010

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  31. [31]

    Causality-invariant interactive mining for cross-modal similarity learning,

    J. Yan, C. Deng, H. Huang, and W. Liu, “Causality-invariant interactive mining for cross-modal similarity learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 9, pp. 6216–6230, 2024

    2024

  32. [32]

    Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks,

    N. Wu, S. Jastrzebski, K. Cho, and K. J. Geras, “Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 24 043–24 055

    2022

  33. [33]

    Pmr: Prototypical modal rebalance for multimodal learning,

    Y . Fan, W. Xu, H. Wang, J. Wang, and S. Guo, “Pmr: Prototypical modal rebalance for multimodal learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 029–20 038

    2023

  34. [34]

    Improving multimodal learning with multi-loss gradient modulation,

    K. Kontras, C. Chatzichristos, M. Blaschko, and M. De V os, “Improving multimodal learning with multi-loss gradient modulation,”arXiv preprint arXiv:2405.07930, 2024

    work page arXiv 2024

  35. [35]

    Mmtm: Multimodal transfer module for cnn fusion,

    H. R. Vaezi Joze, A. Shaban, M. L. Iuzzolino, and K. Koishida, “Mmtm: Multimodal transfer module for cnn fusion,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  36. [36]

    Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations,

    S. Mai, Y . Zeng, and H. Hu, “Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations,”IEEE Transactions on Multimedia, vol. 25, pp. 4121–4134, 2022

    2022

  37. [37]

    Provable dynamic fusion for low-quality multimodal data,

    Q. Zhang, H. Wu, C. Zhang, Q. Hu, H. Fu, J. T. Zhou, and X. Peng, “Provable dynamic fusion for low-quality multimodal data,” inInternational conference on machine learning. PMLR, 2023, pp. 41 753–41 769

    2023

  38. [38]

    Multimodal multi-loss fusion network for sentiment analysis,

    Z. Wu, Z. Gong, J. Koo, and J. Hirschberg, “Multimodal multi-loss fusion network for sentiment analysis,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 3588–3602

    2024

  39. [39]

    Jointly modeling inter-& intra-modality dependencies for multi-modal learning,

    D. Madaan, T. Makino, S. Chopra, and K. Cho, “Jointly modeling inter-& intra-modality dependencies for multi-modal learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 116 084–116 105, 2024

    2024

  40. [40]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,”IEEE transactions on affective computing, vol. 5, no. 4, pp. 377–390, 2014

    2014

  41. [41]

    Look, listen and learn,

    R. Arandjelovic and A. Zisserman, “Look, listen and learn,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 609–617

    2017

  42. [42]

    UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,”arXiv preprint arXiv:1212.0402, 2012

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  43. [43]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,

    A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2236–2246

    2018

  44. [44]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. 12

    2016

  45. [45]

    Multibench: Multiscale benchmarks for multimodal representation learning,

    P. P. Liang, Y . Lyu, X. Fan, Z. Wu, Y . Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y . Zhuet al., “Multibench: Multiscale benchmarks for multimodal representation learning,”arXiv preprint arXiv:2107.07502, 2021

    work page arXiv 2021

  46. [46]

    Moronet: multi-omics integration via graph convolutional networks for biomedical data classification,

    T. Wang, W. Shao, Z. Huang, H. Tang, J. Zhang, Z. Ding, and K. Huang, “Moronet: multi-omics integration via graph convolutional networks for biomedical data classification,”BioRxiv, pp. 2020–07, 2020

    2020

  47. [47]

    A multi-omic atlas of the human frontal cortex for aging and alzheimer’s disease research,

    P. L. De Jager, Y . Ma, C. McCabe, J. Xu, B. N. Vardarajan, D. Felsky, H.-U. Klein, C. C. White, M. A. Peters, B. Lodgsonet al., “A multi-omic atlas of the human frontal cortex for aging and alzheimer’s disease research,”Scientific data, vol. 5, no. 1, pp. 1–13, 2018

    2018

  48. [48]

    Ur-funny: A multimodal language dataset for understanding humor,

    M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morencyet al., “Ur-funny: A multimodal language dataset for understanding humor,”arXiv preprint arXiv:1904.06618, 2019

    work page arXiv 1904

  49. [49]

    Vggsound: A large-scale audio-visual dataset,

    H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “Vggsound: A large-scale audio-visual dataset,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725

    2020

  50. [50]

    Automatic classification and shift detection of facial expressions in event-aware smart environments,

    A. Bernin, L. Müller, S. Ghose, C. Grecos, Q. Wang, R. Jettke, K. von Luck, and F. V ogt, “Automatic classification and shift detection of facial expressions in event-aware smart environments,” inProceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, 2018, pp. 194–201. 13 Appendix In the Supplementary Material, we fir...

    2018

  51. [51]

    This implies that given X, there is no uncertainty about Z, so the conditional entropy H(Z|X) = 0

    Deterministic Encoding: The representation Z is generated by a deterministic encoder ϕ from the multimodal input X, i.e., Z=ϕ(X) . This implies that given X, there is no uncertainty about Z, so the conditional entropy H(Z|X) = 0. This establishes the Markov chainY→X→Z

  52. [52]

    This means the conditional entropy H(C|X, Y) = 0

    Deterministic Interaction: The interaction variable C is a deterministic function of the input X and the target Y , i.e., C=f(X, Y) . This means the conditional entropy H(C|X, Y) = 0 . The variable C is designed to capture specific interaction patterns between modalities inXthat are relevant for predictingY. We begin by expressing∆using a standard informa...

  53. [53]

    Reconstruction Term: Maximizing the reconstruction term, E[logp(z|n, m)] , is equivalent to minimizing the conditional entropy H(Z|V, M) . Since I(Z;V, M) =H(Z)−H(Z|V, M) , this term effectively maximizes the joint mutual information I(Z;V, M) , ensuring that the latent components collectively preserve information about Z

  54. [54]

    Specifically, we have: I(Z;M) =E p(z,m) log p(m|z) p(m) ≤E p(z)[KL(q(m|z)||p(m))], I(Z;N) =E p(z,v) log p(n|z) p(n) ≤E p(z)[KL(q(n|z)||p(n))]

    Regularization Terms: The KL divergence terms serve as variational upper bounds on the mutual information between the representation and the latent components. Specifically, we have: I(Z;M) =E p(z,m) log p(m|z) p(m) ≤E p(z)[KL(q(m|z)||p(m))], I(Z;N) =E p(z,v) log p(n|z) p(n) ≤E p(z)[KL(q(n|z)||p(n))]. (23) Maximizing the ELBO involves minimizing these KL ...

  55. [55]

    Maximizing Reconstruction: The term I(M (1);U (1), R) corresponds to a reconstruction objective. Maximizing it is equivalent to minimizing the conditional entropy H(M (1)|U (1), R), ensuring that the original feature M (1) can be accurately reconstructed from its specific componentU (1) and the shared componentR

  56. [56]

    This is analogous to the KL divergence regularization term in Equation 23

    Maximizing Compactness: The term −I(M (1);U (1)) encourages the specific representation U (1) to be a compact, minimal representation of the information in M (1), following the information bottleneck principle. This is analogous to the KL divergence regularization term in Equation 23

  57. [57]

    Minimizing Redundancy: The term −I(M (2);R|M (1)) aims to minimize the conditional mutual information between M (2) and R given M (1). This encourages R to only contain information that is shared between M (1) andM (2), effectively isolating the redundant (shared) information from the unique aspects of each modality. The third term, the conditional mutual...

    2093