REVIEW 4 major objections 5 minor 2 cited by
I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A mixture of interaction-specialized experts, trained by masking modalities, improves multimodal fusion and makes each prediction's interaction mix interpretable.
desk verdict Solid empirical MoE paper with code and ablations, but the interpretability claims rest on a weakly validated random-masking premise and the 'consistent improvement' claim overreaches on MIMIC. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the interaction expert: a fusion model paired with a prediction head and a designated interaction type from the PID taxonomy. Each expert is trained with a dual objective: a task loss on the weighted ensemble, and an interaction loss computed by perturbing one modality's latent embedding with a random vector and comparing the intact output to the perturbed outputs, using triplet margin loss for uniqueness, cosine similarity for synergy and redundancy, and MSE for regression. The reweighting model, a multilayer perceptron over the latent embeddings, maps expert outputs to the final prediction and doubles as the interpretation mechanism, since its weights are read as sample-level interaction importances and averaged for dataset-level trends.
What would settle it
Train I2MoE on a synthetic two-modality classification task whose true PID components are computable in closed form, then check whether the expert labeled 'synergy' is accurate when synergy is high and at chance when synergy is absent, and whether the learned weights track the known interaction proportions. A cheaper probe is to train a classifier that tries to reconstruct or identify the masked modality from the random-vector-replaced embedding; if it succeeds above chance, the masking does not actually remove the modality, and the weak-supervision assumption fails.
Extended reading notes
Core claim
I2MoE's central claim is that the four PID interaction types can be learned end-to-end by giving each interaction type its own fusion network and a weakly supervised loss, rather than estimating interaction proportions as a separate preprocessing step. In the two-modality case, the intact multimodal output serves as an anchor: each uniqueness expert is pushed to match the intact output only when the other modality is masked, the synergy expert is pushed to be different from both single-modality views, and the redundancy expert is pushed to match both. The reweighting model takes the latent embeddings and outputs soft weights, and the final prediction is the weighted sum of expert predictions. At inference, only one forward pass is needed. The paper presents a human-evaluated local explanation from IMDB where the image-uniqueness and redundancy experts carry the correct genre logit, and dataset-level weight distributions across five benchmarks.
Load-bearing premise
The load-bearing premise, introduced in Section 3.3.2 and supported only by an internal comparison of masking strategies, is that replacing a modality's latent embedding with a random vector truly removes that modality's information, so the interaction losses make each expert learn the intended PID interaction type; if the masked modality leaks, the expert specialization and the interpretation weights are not what they appear to be.
Editorial extensions
If this is right
- I2MoE can be attached to different fusion backbones (MulT, MoE++, SwitchGate, InterpretCC); the paper reports gains on most accuracy/F1/AUROC cells across five datasets, with the main exceptions being accuracy decreases on the imbalanced MIMIC one-year-mortality task alongside AUROC increases.
- Because the reweighting model produces a weight per expert per sample, users get a local explanation of which interaction type drove a decision, and averaging those weights gives a global measure of dataset-level interaction composition.
- Ablations show that removing the interaction loss, applying it to latent embeddings instead of outputs, replacing adaptive weights with a global weight, reducing the number of random perturbations, or dropping uniqueness experts all reduce performance, so each component contributes.
- Random-vector masking outperforms mean-vector and zero-vector masking across most datasets and metrics, supporting the paper's claim that full suppression of the dropped modality is what makes expert specialization work.
- The design scales linearly with the number of modalities, using $n+2$ experts for $n$ modalities, so it does not require a combinatorial number of interaction experts.
Reading between the lines
- Editorial inference: the interpretability claim is about the learned weights, not about ground-truth causal interaction proportions; the paper does not validate expert specialization against datasets with known PID structure, so a synthetic benchmark with analytically known uniqueness, synergy, and redundancy would directly test whether the named experts learn what they claim.
- Editorial inference: because the reweighting model is trained end-to-end with the task loss, high weight on an expert may partly reflect predictive confidence rather than the true proportion of information that modality contributed; users should treat the local weights as model-attribution scores, not as information-theoretic measurements.
- Editorial inference: the extension to more than two modalities assigns one synergy expert and one redundancy expert for all modalities, so interactions that are jointly emergent across three or more modalities are compressed into a single scalar weight; a diagnostic comparing this against pairwise or higher-order interaction estimates would clarify what the multi-modal weights mean.
- Editorial inference: random-vector masking outperforming mean and zero masking may indicate that the model uses the inconsistency of random vectors as a learned 'missing' token; replacing the random vector with a trainable mask token and comparing could isolate whether it is the randomness or the novelty that drives the effect.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes I2MoE, a mixture-of-experts framework for multimodal fusion that uses four interaction experts (two uniqueness experts, a synergy expert, and a redundancy expert) trained with weakly supervised interaction losses based on random-vector modality masking, plus an MLP-based reweighting module that produces sample- and dataset-level importance weights. The authors evaluate I2MoE on two medical and three general datasets, report gains over vanilla fusion baselines when combined with MulT, SwitchGate, MoE++, and InterpretCC, and provide ablations, a qualitative local-interpretability example, a human evaluation, and an analysis of expert agreement/disagreement. The central claims are that I2MoE (i) consistently improves task performance over vanilla fusion backbones and (ii) provides interpretable weights that reflect the PID components of unique, synergistic, and redundant information.
Significance. If the interpretability claim holds, the paper would make a useful contribution by marrying an MoE architecture with PID-motivated interaction modeling and providing sample- and dataset-level attributions. The authors are to be credited for releasing code, performing ablations, and including a human evaluation of the local interpretations. The approach is backbone-agnostic, which broadens its applicability. However, the significance is tempered because the load-bearing premise—that random-vector masking yields faithful 'modality absent' views and that the contrastive losses specialize experts in accordance with PID components—is asserted rather than demonstrated. The task-performance gains are also not uniformly consistent across all reported metrics, and the statistical evidence is limited to mean ± std over three seeds with no significance testing. If the PID-alignment issue were resolved (e.g., through a formal derivation or validation on synthetic data with known ground-truth interactions), the contribution would be much stronger.
major comments (4)
- [Section 3.3.2 / Appendix A] The load-bearing premise of the interpretability claim is that the interaction losses in Section 3.3.2 produce experts that genuinely correspond to the PID components (unique-1, unique-2, synergy, redundancy). The connection to PID in Appendix A is heuristic: Eqs. (5)-(10) use 'proportional to' and 'aligns with' statements rather than derivations. In particular, the assumption that replacing a modality embedding with a random vector (Eqs. (2)-(3)) yields a faithful 'modality absent' view is not guaranteed when the fusion function F is not separable in its inputs, since F(r, E2(x2)) can still carry task-relevant signal through shared fusion or prediction-head parameters trained on intact inputs. The authors acknowledge this assumption in Appendix A but provide neither a formal argument nor a validation that the learned expert specialization matches ground-truth interaction types. Without such validation, the sample-level and dataset-level weights in Section 3.5 may describe the mixture's routing behavior without describing actual unique, synergistic, or redundant information in the data. Please either provide a formal argument (e.g., conditions under which the perturbed views isolate PID components), empirically validate expert specialization on synthetic data with known interaction structure, or explicitly reframe the interpretation claim as describing the routing of the learned mixture rather than a PID decomposition.
- [Appendix B / Table 5] The empirical evidence for random-vector masking only compares task accuracy of Random vs Mean vs Zero masking strategies; it does not verify that the four experts specialize in the intended interaction types. The interaction losses are the only supervision for specialization, and the ablation in Table 4 shows that removing them degrades task performance, but that does not establish that the experts correspond to PID components. Please add an analysis that measures the degree to which each expert's predictions align with the intended interaction type (e.g., on synthetic datasets where interaction components are known, or by measuring whether a 'unique-1' expert indeed stays predictive when modality 2 is masked and degrades when modality 1 is masked).
- [Section 5.2 / Table 2] The abstract claims that I2MoE 'consistently improves task performance,' but Table 2 reports accuracy decreases on MIMIC for all three backbone integrations (-0.56% with SwitchGate, -2.49% with InterpretCC, and -11.82% with MoE++). The explanation in Section 5.2 that this is due to dataset imbalance is plausible but not substantiated; no class-imbalance analysis or confusion-matrix evidence is provided. Moreover, Table 2 does not report error bars or significance tests, so it is unclear whether the positive AUROC changes are meaningful. Please qualify the 'consistently improves' claim to the reported metrics where gains are observed, or provide statistical testing (e.g., paired bootstrap across the three seeds) and a quantitative assessment of the imbalance explanation.
- [Section 6.2 / Table 3] The expert agreement analysis shows that on IMDB, experts disagree in 99.99% of test cases, and in those cases I2MoE is correct only 15.85% of the time (84.14% wrong). This appears to undermine the claim that the mixture-of-experts ensemble robustly leverages diverse expert opinions in that dataset, and it is in tension with the broader claim (Section 6.1, Figure 5) that I2MoE 'consistently surpasses' every individual expert. Since Figure 5 reports average accuracy, it is possible for the average to be better while the mixture is far worse than a single expert in the majority of cases. Please reconcile these observations, e.g., by reporting per-expert accuracy in the disagreement regime and discussing when the ensemble actually helps versus when a single expert or a different aggregation rule would be preferable.
minor comments (5)
- [Appendix D] There is a typo in 'Numer of modalities' (should be 'Number of modalities').
- [Figure 2] The abbreviations 'Syngy' and 'Rdncy' in Figure 2 are unclear; please use full words or standard abbreviations in the figure.
- [Section 3.4] The description of the uniqueness interaction loss for n modalities says 'The masked modality i serves as a negative example, while all other perturbed inputs are treated as positive examples.' This is consistent with Appendix C's formula, but the wording 'all other perturbed inputs' is ambiguous about whether the anchor is included; please clarify.
- [Section 5.3 / Appendix H] The human evaluation asks participants whether the expert weights 'make sense'; this is a subjective reasonableness check and does not validate whether the weights correspond to actual unique/synergy/redundant information. This is acceptable as a usability check, but it should be clearly labeled as such rather than as validation of PID alignment.
- [Appendix C] The synergy loss in Appendix C uses CosSim between the full output and each perturbed output, while the redundancy loss uses (1 - CosSim). In Section 3.3.2, the text says for synergy the outputs with masked modality are negative examples (so the loss should be maximized), but the Appendix formula as written does not indicate a sign flip for the synergy expert. Please ensure the notation is unambiguous about whether the loss is to be minimized or maximized and whether a negative sign is included.
Circularity Check
No circularity found: I2MoE's accuracy gains are benchmarked against external baselines, and its interpretability weights are model-internal outputs rather than fitted quantities relabeled as independent predictions.
full rationale
The paper's primary claims are empirical: I2MoE is evaluated against external fusion baselines (MulT, SwitchGate, MoE++, InterpretCC, etc.) on five datasets, with encoders and prediction heads held fixed (Section 4, Tables 1-2), so the reported accuracy/AUROC/F1 gains are not reducible to a fitted parameter renamed as a prediction. The interpretability output (Section 3.5) is a summary of the trained reweighting model W's weights together with each expert's prediction; the paper does not claim to predict an independently measured PID ground truth, so there is no equation-level equivalence between an input quantity and the reported interpretation. The interaction losses (Section 3.3.2, Appendix C) do impose the intended contrastive relationships on the experts by construction, but the learned weights are free parameters optimized by the task loss, and the paper presents them as model-based explanation rather than as a derivation of PID from first principles. The random-vector masking assumption (Eqs. 2-3; 'Assuming ~Xi contains no task-relevant information', Appendix A) is a correctness/validity risk—if masking does not suppress the modality, the expert labels are not faithful—but this is not a circular argument, because no result is defined in terms of the quantity it is claimed to explain. The only overlap-authors citation (Flex-MoE, Yun et al. 2024) is used for ADNI/MIMIC preprocessing (Appendix E) and is not load-bearing for the architecture, loss, or evaluation; per the rules, a non-load-bearing self-citation does not raise the circularity score. No pattern from the enumerated list (self-definitional reduction, fitted-input-as-prediction, self-citation chain, uniqueness import, ansatz-by-citation, or renaming) is present at the level of a specific equation or claimed derivation.
Assumptions & free parameters
free parameters (3)
- interaction_loss_weight_lambda_int =
Per-dataset values: ADNI 0.5, MIMIC 0.01, IMDB 0.5, MOSI 0.005, ENRICO 0.5 for I2MoE-MulT; other values for other…
- reweighting_temperature =
1 to 4 depending on dataset and backbone
- reweighting_MLP_hidden_dim_and_layers =
Hidden dim 128-256, layers 1-3 across configurations
assumptions (4)
- domain assumption PID decomposition into unique, synergistic, redundant, and redundant information is an appropriate and complete description of two-modality interactions
- ad hoc to paper Replacing a modality embedding with a random vector fully suppresses that modality's task-relevant information
- ad hoc to paper The interaction losses in Appendix C are a valid proxy for the PID components (Eqs. 5-10)
- domain assumption The reweighting model weights w_i reflect the true contribution of each interaction type to the prediction
Cite this review
Pith. "Pith review of I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts." pith.science (2026). https://pith.science/paper/RPTMHUH5
@misc{pith2026250519190,
author = {Pith},
title = {Pith review of: I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts},
year = {2026},
howpublished = {\url{https://pith.science/paper/RPTMHUH5}},
note = {Machine review of arXiv:2505.19190}
}
read the original abstract
Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 2 Pith papers
-
BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure
BrainFIBRE pretrains a five-expert Mixture-of-Experts model on NODDI-derived microstructural maps and outperforms prior deep models on age, sex, cerebrovascular, neurodegenerative, and cognitive prediction.
-
iLENS: Interpretable LLM-Guided Mixture-of-Experts for Neuroimaging Survival Analysis
LLM-guided mixture-of-experts yields competitive C-index and stronger LogRank subtype separation for MCI-to-AD conversion on ADNI neuroimaging plus clinical notes.
Reference graph
Works this paper leans on
-
[1]
Arevalo, J., Solorio, T., Montes-y G´omez, M., and Gonz´alez, F. A. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992,
-
[6]
Comparison of MulT and I2MoE-MulT on training time, inference latency, and model size across datasets. Train / epoch (s) Inference (s) # Params Dataset Modalities MulT I 2 MoE-MulT MulT I 2 MoE-MulT MulT I 2 MoE-MulT ADNI I, G, C, B 8.98 ± 0.04 16.82 ± 0.02 1.34 ± 0.00 2.29 ± 0.00 1,072,131 6,696,728 MIMIC L, N, C 2.24 ± 0.01 33.67 ± 0.67 0.15 ± 0.00 0.91...
work page 2024
-
[11]
V ., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A
Lin, X. V ., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts. arXiv preprint arXiv:2407.21770,
-
[12]
Liu, Z., Shen, Y ., Lakshminarasimhan, V . B., Liang, P. P., Zadeh, A., and Morency, L.-P. Efficient low-rank mul- timodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064,
-
[13]
P., Deng, Z., Salakhutdinov, R., and Morency, L.-P
Lyu, Y ., Liang, P. P., Deng, Z., Salakhutdinov, R., and Morency, L.-P. Dime: Fine-grained interpretations of multimodal models via disentangled local explanations. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 455–467,
work page 2022
-
[16]
Tsai, Y .-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceed- ings of the conference. Association for computational linguistics. Meeting, volume 2019, pp
work page 2019
-
[17]
Tsai, Y .-H. H., Ma, M. Q., Yang, M., Salakhutdinov, R., and Morency, L.-P. Multimodal routing: Improving local and global interpretability of multimodal language analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empir- ical Methods in Natural Language Processing, volume 2020, pp
work page 2020
-
[18]
Measuring Cross-Modal Interactions in Multimodal Models
Wenderoth, L., Hemker, K., Simidjievski, N., and Jamnik, M. Measuring cross-modal interactions in multimodal models. arXiv preprint arXiv:2412.15828,
Show all 26 references
-
[19]
Williams, P. L. and Beer, R. D. Nonnegative decom- position of multivariate information. arXiv preprint arXiv:1004.2515,
-
[20]
Beyond additive fusion: Learning non-additive multimodal interactions
W¨ortwein, T., Sheeber, L., Allen, N., Cohn, J., and Morency, L.-P. Beyond additive fusion: Learning non-additive multimodal interactions. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4681– 4696,
2022
-
[21]
URL https://arxiv.org/abs/2204. 00102. Yu, H., Qi, Z., Jang, L., Salakhutdinov, R., Morency, L.-P., and Liang, P. P. Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Langu...
2024
-
[23]
Mosi: multimodal corpus of sentiment intensity and subjectiv- ity analysis in online opinion videos
Zadeh, A., Zellers, R., Pincus, E., and Morency, L.-P. Mosi: multimodal corpus of sentiment intensity and subjectiv- ity analysis in online opinion videos. arXiv preprint arXiv:1606.06259,
-
[24]
The Connection between Interaction Loss and PID We link our perturbation-based losses to components in Partial Information Decomposition (PID), following Bertschinger et al
12 Interpretable Multimodal Interaction-aware Mixture-of-Experts A. The Connection between Interaction Loss and PID We link our perturbation-based losses to components in Partial Information Decomposition (PID), following Bertschinger et al. (2014): I(T ; X1, X2) = Red(T ; X1,...
2014
-
[26]
For the genomics, clinical, and biospecimen modalities, we used a one-hidden-layer MLP with a hidden dimension of 256 as the encoder
with a hidden dimension of 256 as the encoder. For the genomics, clinical, and biospecimen modalities, we used a one-hidden-layer MLP with a hidden dimension of 256 as the encoder. ❷ MIMIC Dataset: For all lab, note, and code modalities, we utilized an LSTM with a hidden dimen...
2021
-
[1991]
Moe++: Accelerat- ing mixture-of-experts methods with zero-computation experts
Jin, P., Zhu, B., Yuan, L., and Yan, S. Moe++: Accelerat- ing mixture-of-experts methods with zero-computation experts. arXiv preprint arXiv:2410.07348,
-
[1999]
C., Li `o, P., and Simidjievski, N
Dominici, G., Barbiero, P., Magister, L. C., Li `o, P., and Simidjievski, N. Sharcs: Shared concept space for explainable multimodal learning. arXiv preprint arXiv:2307.00316,
-
[2000]
A., Arik, S
Ismail, A. A., Arik, S. ¨O., Yoon, J., Taly, A., Feizi, S., and Pfister, T. Interpretable mixture of experts for structured data. arXiv preprint arXiv:2206.02107,
-
[2012]
Yun, S., Choi, I., Peng, J., Wu, Y ., Bao, J., Zhang, Q., Xin, J., Long, Q., and Chen, T
doi: 10.1109/TNNLS.2012.2200299. Yun, S., Choi, I., Peng, J., Wu, Y ., Bao, J., Zhang, Q., Xin, J., Long, Q., and Chen, T. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. arXiv preprint arXiv:2410.08245,
2012
-
[2016]
What to align in multimodal contrastive learning? arXiv preprint arXiv:2409.07402,
9 Interpretable Multimodal Interaction-aware Mixture-of-Experts Dufumier, B., Castillo-Navarro, J., Tuia, D., and Thiran, J.-P. What to align in multimodal contrastive learning? arXiv preprint arXiv:2409.07402,
-
[2017]
A., Jaggi, M., and K ¨aser, T
Swamy, V ., Montariol, S., Blackwell, J., Frej, J. A., Jaggi, M., and K ¨aser, T. Interpretcc: Intrinsic user-centric in- terpretability through global mixture of experts. arXiv preprint arXiv:2402.02933, 2024a. Swamy, V ., Satayeva, M., Frej, J., Bossy, T., V ogels, T., Jaggi...
-
[2018]
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,
-
[2019]
ieee.org/document/8269806
URL https://ieeexplore. ieee.org/document/8269806. Barnum, G., Talukder, S., and Yue, Y . On the benefits of early fusion in multimodal representation learning,
-
[2020]
Bertschinger, N., Rauh, J., Olbrich, E., Jost, J., and Ay, N
URL https://arxiv.org/abs/2011.07191. Bertschinger, N., Rauh, J., Olbrich, E., Jost, J., and Ay, N. Quantifying unique information. Entropy, 16(4):2161– 2183,
2011 arXiv
-
[2021]
P., Lyu, Y ., Chhablani, G., Jain, N., Deng, Z., Wang, X., Morency, L.-P., and Salakhutdinov, R
Liang, P. P., Lyu, Y ., Chhablani, G., Jain, N., Deng, Z., Wang, X., Morency, L.-P., and Salakhutdinov, R. Multi- viz: Towards visualizing and understanding multimodal models. arXiv preprint arXiv:2207.00056, 2022a. Liang, P. P., Zadeh, A., and Morency, L.-P. Founda- tions and...
-
[2023]
org/abs/2302.12247
URL https://arxiv. org/abs/2302.12247. Liang, P. P., Deng, Z., Ma, M. Q., Zou, J. Y ., Morency, L.-P., and Salakhutdinov, R. Factorized contrastive learn- ing: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36,
-
[2024]
I., Pohl, K
Esmaeilzadeh, S., Belivanis, D. I., Pohl, K. M., and Adeli, E. End-to-end alzheimer’s disease diagnosis and biomarker identification. In Machine Learning in Medical Imaging: 9th International Workshop, MLMI 2018, Held in Con- junction with MICCAI 2018, Granada, Spain, Septembe...
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.