REVIEW 4 cited by
Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-modal network, which is counter-intuitive since multiple signals generally bring more information. This work provides a theoretical explanation for the emergence of such performance gap in neural networks for the prevalent joint training framework. Based on a simplified data distribution that captures the realistic property of multi-modal data, we prove that for the multi-modal late-fusion network with (smoothed) ReLU activation trained jointly by gradient descent, different modalities will compete with each other. The encoder networks will learn only a subset of modalities. We refer to this phenomenon as modality competition. The losing modalities, which fail to be discovered, are the origins where the sub-optimality of joint training comes from. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.
Forward citations
Cited by 4 Pith papers
-
Information-Theoretic Decomposition for Multimodal Interaction Learning
DMIL is a multimodal learning framework that decomposes sample-specific interactions into redundant, unique, and synergistic components via variational architecture and uses them for adaptive fine-tuning.
-
Boosting Multimodal Federated Learning via Chained Modality Optimization
FedMChain improves multimodal federated learning by chaining modality-wise optimization phases with error-compensated regularization and sparse sign-guided aggregation to mitigate modality competition and cut communic...
-
Optimizing Deep Learning Photometric Redshifts for the Roman Space Telescope with HST/CANDELS
PITA, a new semi-supervised deep learning algorithm, outperforms prior photo-z methods by using a triple-task loss on images, colors, and available redshifts to produce a smooth latent space.
-
Foundation Models for Astrophysics
Astronomical 'foundation models' largely reuse transformers and self-supervised pretraining, but evidence of transfer to new instruments, populations, or tasks remains rare; the paper argues such evidence, not archite...
Discussion (0). Sign in to comment.