Pith. sign in

REVIEW 3 major objections 6 minor 65 references

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that frozen pretrained unimodal encoders can be adapted to multimodal tasks by learning low-rank subspaces that separate shared from modality-specific information, without heavy paired-data training.

desk verdict A real combination of LoReFT and multimodal decoupling with solid ablations, but the whole edifice rests on a linear-shared-subspace assumption the paper never tests. read the letter →

arxiv 2607.16789 v1 pith:VEWLFEWC submitted 2026-07-18 cs.LG

Sana Tonekaboni , Viktoria Schuster , Caroline Uhler This is my paper · ORCID
classification cs.LG
keywords multimodalrepresentationlearninglow-rankfine-tuningdisentangledrepresentationssharedinformationmodality-specificmissingmodalityrobustnessHSICindependenceInfoNCE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiLoReFT claims that a pair of frozen, pretrained unimodal encoders can be adapted to multimodal tasks by learning only a few low-rank projection subspaces: one shared across modalities and one private to each modality. Each fine-tuned representation is the original embedding plus a low-rank edit computed in those subspaces, and training is self-supervised, guided by three structural objectives—statistical independence between shared and private components, orthogonality between subspaces, and preservation of the original embedding's information. If the claim holds, multimodal systems can reuse strong unimodal models without large aligned paired datasets or full fine-tuning, and the resulting representations show explicitly where shared versus modality-specific information lives. The paper supports the claim with synthetic data whose generative factors are known and with image-caption, audiovisual-emotion, and humor benchmarks, including a missing-modality setting.

What carries the argument

The load-bearing object is the triple of low-rank projection matrices: Rs shared across modalities, Rm1 and Rm2 private, each paired with a small nonlinear transform f acting in the projected subspace. Representation fine-tuning edits each frozen embedding by adding back the difference between the transformed and unedited projections, so all trainable parameters live in low-rank spaces. The decoupling work is done by the three losses: HSIC removes nonlinear statistical dependence between shared and private components, the orthogonality loss prevents geometric overlap of the projection bases, and the InfoNCE loss ties each component to its original embedding while aligning shared components a

What would settle it

Build a synthetic pair of modalities in which the shared factor is a nonlinear function of the pretrained embedding coordinates (for example, an XOR of two coordinates or a high-frequency sinusoid) while private factors stay linear; train MultiLoReFT and probe zs, zm1, zm2 with a flexible nonlinear classifier for the shared label. If the shared label is recoverable from a private component, or the shared component cannot separate it after rank pruning, the claim that one shared low-rank projection suffices to isolate common information is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that shared and modality-specific information can be decoupled by editing pretrained unimodal representations in low-rank subspaces. MultiLoReFT defines a shared low-rank projection Rs applied to both modalities and private projections Rm1 and Rm2 for the two modalities; each fine-tuned representation is the original embedding plus a correction built by projecting into the subspace, passing through a small nonlinear transform, and projecting the difference back (Phi_i = h_i + R_s^T(f_si(h_i) - R_s h_i) + R_mi^T(f_mi(h_i) - R_mi h_i)). The edits are learned without task labels by minimizing a composite objective: HSIC, a nonparametric independence measure, m

Load-bearing premise

The load-bearing premise is that the information shared by two modalities lies in one common low-dimensional linear subspace of the pretrained representations, so a single low-rank matrix Rs applied to both embeddings can expose it; if the cross-modal structure is nonlinear, or the two encoders cannot be aligned by a single linear map, the disentanglement and missing-modality guarantees lose their foundation.

Editorial extensions

If this is right

  • If the claim is right, multimodal fine-tuning requires training only the projection matrices and small transforms, with a parameter cost tied to subspace rank rather than encoder size.
  • The shared projection becomes a reusable cross-modal code: either modality can be mapped into it for retrieval or downstream tasks, and the private projections expose what each modality contributes uniquely.
  • When one modality disappears at test time, the remaining modality's edited representation should beat its pretrained embedding, since training has forced the shared component to preserve cross-modal information.
  • The decoupling is inspectable with simple probes: a linear regressor on each component reveals which generative factor lives where, giving a way to audit leakage in any fusion model.
  • The framework extends to more than two modalities by adding private subspaces, though the paper states that partially shared factors become harder to interpret and that performance inherits the limits of the frozen unimodal encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the shared subspace is transfer: if Rs captures modality-agnostic semantics, it should transfer across tasks or datasets, while private subspaces should not; this is not tested in the paper.
  • The linear-subspace assumption could be stress-tested with synthetic data where the common factor is a nonlinear function of the pretrained embeddings; if decoupling gaps collapse, the shared projection would need to be nonlinear rather than a single Rs.
  • The real-data decoupling evaluation leans on proxy attribute labels (Race/Age assigned to video, Sentence ID to audio); a stricter audit would use ground-truth generative factors or nonlinear probes on all components to rule out leakage.
  • If the missing-modality result holds broadly, it implies a deployment protocol for sensor dropout: keep frozen encoders plus learned projections and fall back to the shared component whenever one stream fails.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MultiLoReFT, a parameter-efficient fine-tuning method for multimodal learning with two frozen unimodal encoders. It learns low-rank projection matrices R_s, R_m1, R_m2 and small nonlinear transforms f_{s i}, f_{m i} that edit the pretrained embeddings h_1, h_2 (Eqs. 3-4) so that shared and modality-specific projections (Eqs. 1-2) are statistically independent, orthogonal, and mutually informative. The training objective (Eq. 5) combines an HSIC independence loss (Eq. 6), an orthogonality loss (Eq. 7), and an InfoNCE-style cross-modal mutual-information loss (Eq. 8), together with an adaptive rank-pruning mechanism (Algorithm 1). Experiments on two simulations with known generative factors and on Flickr30K-Multi, Crema-D, and UR-FUNNY evaluate information decoupling, downstream multimodal prediction, and robustness to missing modalities. The central claim is that MultiLoReFT learns interpretable projection subspaces that decouple shared and modality-specific information while remaining parameter-efficient.

Significance. If the decoupling claim is substantiated, the method would be a practical bridge between parameter-efficient fine-tuning and interpretable multimodal fusion, since it adapts frozen unimodal encoders with a small parameter overhead and yields a direct handle on shared versus private factors. The paper has concrete strengths: the formulation is explicit and reproducible in principle (code is promised), the ablations (Table 4) show that each loss term and the pruning mechanism contribute, and the synthetic experiments use ground-truth generative factors. However, the evidence for the central claim is currently weaker than the abstract suggests. In particular, the absolute leakage of the shared label into private subspaces is high, the synthetic data are generated from linear mixtures, and the cross-modal behavior of the shared subspace is optimized by construction through the InfoNCE loss. These gaps do not invalidate the approach, but they require re-basing the decoupling claim on stronger and more external evidence.

major comments (3)
  1. [Section 5.1, Table 1 (Simulation I)] The reported Delta metrics hide substantial absolute leakage. For the shared label, z_m1 achieves 82.0% and z_m2 achieves 61.5% accuracy, while chance is 25% for a 4-class label. This means the shared factor can be decoded to a large degree from the 'modality-specific' subspaces, which directly contradicts the claim that shared information is isolated in z_s. The same issue appears in Simulation II, where z_m2 predicts the shared label at 61.7% (binary chance 50%). Please report absolute accuracies relative to chance, and ideally a formal conditional-independence test, for each component. The current presentation overstates the degree of decoupling because it emphasizes Delta rather than the remaining leakage.
  2. [Section 3.1, Eqs. 1-4; Appendix A.1.2] The disentanglement claim is tied to a linear low-rank assumption. The synthetic data are generated as h_1 = [z_m1, z_s] W_1 and h_2 = [z_m2, z_s] W_2, so the shared factor occupies a linear subspace by construction and is therefore accessible to a linear R_s. Real pretrained embeddings may encode shared semantics in a nonlinear or geometrically mismatched way. Since the edit in Eqs. 3-4 only moves h_i within the row space of R_s and R_{m i}, any shared structure that is not linearly recoverable from h_i is discarded. Please add a synthetic experiment with nonlinear mixing (e.g., h_i = g_i([z_s, z_m i]) with a multilayer g_i) and, if possible, a real-data sanity check in which a known shared factor is nonlinearly related to both modalities. The 'decoupling' claim should be limited to linearly accessible shared structure unless such evidence is provided.
  3. [Section 3.2, Eq. 8; Section 5.1] The cross-modal property of the shared subspace is partially enforced by construction. In Eq. 8, z(i) is formed by concatenating the modality-specific projection of modality i with the shared projection of the opposite modality, so the InfoNCE objective directly trains z_s from one modality to be predictive of the other modality's embedding. The evaluation then shows that z_s is cross-modal. This is to some extent a check that the objective was optimized, rather than a test that the factorization matches an external ground-truth generative structure. Because the simulated ground truth is generated from the same shared/private decomposition, it provides a favorable test bed. I recommend adding an evaluation that does not rely on this construction, such as an intervention test that removes the learned shared subspace and measures whether the residual still predicts the shared label, or a r
minor comments (6)
  1. [Table 2] The Flickr30 block has inconsistent row labels: the DRIM-U row lists 'zm2(Audio)' but the column is Text, and the APOLLO row repeats 'zm1(Text)' twice. Also, the dataset name should be 'Flickr30K-Multi' rather than 'Flickr30'.
  2. [Table 1] The Simulation II block is labeled only for Shared and M1, while the text and legend refer to Modality II labels as well. Either report the M2 column or remove the reference.
  3. [Table 4] The row 'MultiLoReFT - Independence' contains a formatting/typo error: '0.71.9±0.15.6'. Please correct this entry.
  4. [Figure 4] The missing-modality results are reported only as a bar chart without error bars, numerical values, or the number of seeds. Please include the underlying numbers and error bars so that the 'consistently improve' claim can be assessed.
  5. [Section 3.2 / 3.4] The relationship between the weights lambda in Eq. 5 and the GradNorm mechanism in Algorithm 2 is unclear. If GradNorm replaces the lambda weights, this should be stated explicitly; if not, the interaction should be described.
  6. [Section 3.2] The fixed non-trainable random projection used to handle differing dimensions in the MI loss is described only in prose. Please specify its distribution, dimensionality, and whether the same seed is used across all runs, for reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: decoupling is imposed via Eqs. 1-8 and validated on external labels; InfoNCE's opposite-modality construction is an explicit design, not a hidden fit.

full rationale

The derivation chain is self-contained. Eqs. 1-4 are explicit design choices (a shared low-rank projection Rs and modality-private projections Rm1,Rm2), not results derived from the evaluation. The objectives in Eqs. 5-8 are training objectives: HSIC and orthogonality explicitly enforce disentanglement, and the InfoNCE term deliberately constructs z(i) using the opposite modality's shared projection to enforce cross-modal consistency. The paper states this design intent rather than presenting it as an emergent prediction, and the downstream decoupling evaluation uses external labels (simulated generative factors, Crema-D metadata, Flickr language labels) that are not used in training; hence the reported gaps are not the training loss renamed. The simulation studies are model-matched (shared latent zs is linearly embedded in both h1 and h2 in App. A.1), which is a controlled validation of the assumptions, not a circular reduction: baselines on the same paired data fail to match the decoupling gaps, so the learned subspaces are not forced by the data construction alone. No load-bearing self-citation or uniqueness theorem is invoked. The honest limitations in the Discussion (performance bounded by pretrained encoders) and A.6 (no ground-truth disentanglement labels for CMU-MOSI) further indicate the claims are not presented as self-proving. Score 1 reflects only the mild definitional flavor of the InfoNCE objective/evaluation alignment; there is no significant circularity.

Assumptions & free parameters 4 free parameters · 7 assumptions · 1 invented entities

No physical entities are introduced; the ledger entries are the linear-subspace constructs and the hand-set or unreported constants that the central decoupling claim rests on. The two domain assumptions — linear accessibility of shared structure and proxy-label attributions — are the ones most likely to fail on new data.

free parameters (4)
  • Pruning threshold ε = not reported
    Algorithm 1 and 2 require ε to decide which singular directions are pruned; no value or schedule is given in the main text or appendix, so the reported results depend on an unreported hand-set constant.
  • InfoNCE temperature τ = not reported
    Eq. 8 uses τ to control the softmax sharpness; the value is never reported despite affecting the MI loss balance.
  • Initial subspace ranks r0 = d (10 for simulations, 700 for Crema-D/Flickr)
    Subspaces are initialized at the unimodal representation dimension (§5.1); the choice is made, not fitted, but it co-determines the reported results.
  • Training-schedule constants = warm-up 50 epochs, patience 150, cap 10%, tolerance 1%
    Warm-up length, convergence patience, pruning cap, and validation-MI tolerance are hand-set and jointly determine the converged ranks in Table 5.
assumptions (7)
  • standard math HSIC with characteristic RBF kernels exactly characterizes statistical independence (Gretton et al. 2007)
    Used in Eq. 6 for Lindep; accepted background, but the unbiased estimator in mini-batches introduces an approximation.
  • standard math InfoNCE is a valid lower bound on mutual information
    Invoked after Eq. 8 to justify LMI; no citation to the original InfoNCE work is given in the paper.
  • domain assumption Pretrained unimodal representations share the same dimension d so that Rs, Rm1, Rm2 ∈ R^{r×d} are well-defined
    Eqs. 1–4 assume h1, h2 ∈ R^d; the paper only handles mismatched dimensions later via a fixed random projection head for the MI loss (§3.2).
  • domain assumption Shared cross-modal information lies in a common low-dimensional linear subspace of pretrained embeddings
    The whole framework (Eqs. 1–4) is a sum of low-rank projections; if the shared structure is not linearly accessible, decoupling cannot succeed.
  • domain assumption Orthogonality plus pairwise HSIC independence yields the intended interpretable decoupling
    §3.2 argues one constraint alone is insufficient; the claim that the combination separates truly shared from correlated modality-specific noise is a modeling premise, partially supported by the ablations.
  • domain assumption Proxy-label attributions on real data are correct (Sentence ID from audio; Age/Race from video; Language from text)
    Table 2 validation treats Crema-D metadata and Flickr language as ground truth for which modality carries which factor; wrong attributions would invalidate the decoupling gaps.
  • domain assumption Fixed random projection head preserves enough information for the InfoNCE estimate
    §3.2 projects h_i and z(i) into a common random space; the sufficiency of this non-trainable head is assumed, not analyzed.
invented entities (1)
  • Learned shared/modality-specific low-rank subspaces (Rs, Rm1, Rm2)
    purpose: Representational constructs that supposedly isolate cross-modal shared content and unimodal-only content inside frozen encoder representations
    No falsifiable handle outside the paper; they are validated only through the paper's own evaluation protocol (proxy labels, nearest-neighbor visualizations, missing-modality probes).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning." pith.science (2026). https://pith.science/paper/VEWLFEWC

@misc{pith2026260716789,
  author       = {Pith},
  title        = {Pith review of: MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VEWLFEWC}},
  note         = {Machine review of arXiv:2607.16789}
}
read the original abstract

Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities. However, training multimodal models faces two main obstacles. First, collecting large-scale, well-aligned paired multimodal datasets is often impractical, making end-to-end multimodal training difficult. Second, existing multimodal representations frequently entangle information shared across modalities with modality-specific information, hindering interpretability and control. We introduce MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models. MultiLoReFT extends low-rank adaptation to the multimodal setting and learns interpretable projection subspaces that decouple shared and modality-specific information. Across simulated and real-world benchmarks, it produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.

Figures

Figures reproduced from arXiv: 2607.16789 by the authors.

Figure 1
Figure 1. Overview of MultiLoReFT. Pretrained unimodal encoders (with frozen parameters) produce representations h1 and h2, which are fine-tuned through low-rank projections into shared (Rs) and modality-specific (Rm1, Rm2) subspaces. Nonlinear transforms (fs1, fs2, fm1, fm2) learn the representation edit in the lower-rank space that needs to be subtracted from the projection of the representa￾tions. Once edited in the low-ra… view at source ↗
Figure 2
Figure 2. Visualization of subspaces learned by MultiLoReFT on Simulation I. Each panel shows a 2D PCA projection of the shared or modality-specific representations, colored by the under￾lying generative label. For the shared label (left columns), clear clustering emerges in the shared subspace while modality-specific subspaces remain unstructured. For the modality I and II labels, the corresponding modality-specific subspace… view at source ↗
Figure 3
Figure 3. (a) PCA visualizations of MultiLoReFT representations on Crema-D and Flickr30. Task-relevant information is concentrated in the corresponding modality-specific components, while other components remain closer to random structure, illustrating effective disentanglement. (b) An example image from Flickt30 dataset with its closest samples in the image-specific subspace and shared subspace. dimension, and during trainin… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Downstream classification accuracy under missing￾modality inference. For each dataset, we compare the pretrained representation h and the fine-tuned representation Φ of the avail￾able modality when the other modality is missing. “M1 missing” evaluates prediction using …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 3 linked inside Pith

  1. [1]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    Multimodal machine learning: A survey and taxonomy , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2019 , publisher=

    2019

  2. [2]

    Nature communications , volume =

    Cross-modal autoencoder framework learns holistic representations of cardiovascular state , author =. Nature communications , volume =

  3. [3]

    Proceedings of the IEEE , volume =

    Machine learning approaches to single-cell data integration and translation , author =. Proceedings of the IEEE , volume =

  4. [4]

    Nature Communications , volume =

    Multi-domain translation between single-cell imaging and sequencing data using autoencoders , author =. Nature Communications , volume =

  5. [5]

    Nature Communications , volume =

    Graph-based autoencoder integrates spatial transcriptomics with chromatin images and identifies joint biomarkers for Alzheimer’s disease , author =. Nature Communications , volume =

  6. [6]

    Advances in Neural Information Processing Systems , volume =

    Unpaired multi-domain causal representation learning , author =. Advances in Neural Information Processing Systems , volume =

  7. [7]

    Proceedings of the International Conference on Machine Learning (ICML) , year=

    Learning transferable visual models from natural language supervision , author=. Proceedings of the International Conference on Machine Learning (ICML) , year=

  8. [8]

    International Conference on Machine Learning (ICML) , year=

    BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation , author=. International Conference on Machine Learning (ICML) , year=

Show all 65 references
  1. [9]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Mvitv2: Improved multiscale vision transformers for classification and detection , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  2. [10]

    IEEE Journal of Selected Topics in Signal Processing , volume=

    WavLM: Large-scale self-supervised pre-training for full-stack speech processing , author=. IEEE Journal of Selected Topics in Signal Processing , volume=. 2022 , publisher=

    2022

  3. [11]

    Nature machine intelligence , volume=

    Parameter-efficient fine-tuning of large-scale pre-trained language models , author=. Nature machine intelligence , volume=. 2023 , publisher=

    2023

  4. [12]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , month =

    Lee, Mihee and Pavlovic, Vladimir , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , month =. 2021 , pages =

    2021

  5. [13]

    In 2021 IEEE , author=

    Multimodal vae active inference controller. In 2021 IEEE , author=. RSJ International Conference on Intelligent Robots and Systems (IROS) , pages=

    2021

  6. [14]

    arXiv preprint arXiv:1606.06259 , year=

    Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos , author=. arXiv preprint arXiv:1606.06259 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2405.13980 , year=

    Rank Reduction Autoencoders , author=. arXiv preprint arXiv:2405.13980 , year=

    arXiv

  8. [16]

    Advances in neural information processing systems , volume=

    Flamingo: a visual language model for few-shot learning , author=. Advances in neural information processing systems , volume=

  9. [17]

    UR - FUNNY : A Multimodal Language Dataset for Understanding Humor

    Hasan, Md Kamrul and Rahman, Wasifur and Bagher Zadeh, AmirAli and Zhong, Jianyuan and Tanveer, Md Iftekhar and Morency, Louis-Philippe and Hoque, Mohammed (Ehsan). UR - FUNNY : A Multimodal Language Dataset for Understanding Humor. Proceedings of the 2019 Conference on Empiri...

    2019 doi

  10. [18]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , volume=

  11. [19]

    arXiv preprint arXiv:2106.09685 , year=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. arXiv preprint arXiv:2106.09685 , year=

    arXiv

  12. [20]

    npj Digital Medicine , volume=

    A Representation Fusion Framework for Decoupling Diagnostic Information in Multimodal Learning , author=. npj Digital Medicine , volume=. 2025 , publisher=

    2025

  13. [21]

    The Thirteenth International Conference on Learning Representations , year=

    An Information Criterion for Controlled Disentanglement of Multimodal Data , author=. The Thirteenth International Conference on Learning Representations , year=

  14. [22]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  15. [23]

    ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    A pre-trained audio-visual transformer for emotion recognition , author=. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2022 , organization=

    2022

  16. [24]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    MMPareto: boosting multimodal learning with innocent unimodal assistance , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

  17. [25]

    2019 8th international conference on affective computing and intelligent interaction (ACII) , pages=

    Multimodal and temporal perception of audio-visual cues for emotion recognition , author=. 2019 8th international conference on affective computing and intelligent interaction (ACII) , pages=. 2019 , organization=

    2019

  18. [26]

    Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval , pages=

    C-pack: Packed resources for general chinese embeddings , author=. Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval , pages=

  19. [27]

    Advances in Neural Information Processing Systems , volume=

    Reft: Representation finetuning for language models , author=. Advances in Neural Information Processing Systems , volume=

  20. [28]

    Advances in neural information processing systems , volume=

    A kernel statistical test of independence , author=. Advances in neural information processing systems , volume=

  21. [29]

    IEEE Access , volume=

    Simple and effective multimodal learning based on pre-trained transformer models , author=. IEEE Access , volume=. 2022 , publisher=

    2022

  22. [30]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Emerging properties in self-supervised vision transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  23. [31]

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages=

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages=

    2019

  24. [32]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Language-agnostic BERT Sentence Embedding , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  25. [33]

    Advances in neural information processing systems , volume=

    wav2vec 2.0: A framework for self-supervised learning of speech representations , author=. Advances in neural information processing systems , volume=

  26. [34]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages=

    The Kinetics Human Action Video Dataset , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages=

  27. [35]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

    Deep Residual Learning for Image Recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages=

  28. [36]

    Learning Factorized Multimodal Representations , author=

  29. [37]

    IEEE transactions on affective computing , volume=

    Crema-d: Crowd-sourced emotional multimodal actors dataset , author=. IEEE transactions on affective computing , volume=. 2014 , publisher=

    2014

  30. [38]

    International conference on learning representations , year=

    Multiplicative interactions and where to find them , author=. International conference on learning representations , year=

  31. [39]

    International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=

    Drim: Learning disentangled representations from incomplete multimodal healthcare data , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2024 , organization=

    2024

  32. [40]

    Nature Computational Science , pages=

    Partially shared multi-modal embedding learns holistic representation of cell state , author=. Nature Computational Science , pages=. 2026 , publisher=

    2026

  33. [41]

    European Conference on Computer Vision , pages=

    Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  34. [42]

    2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages=

    Multi-Modal Large Language Models are Effective Vision Learners , author=. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages=. 2025 , organization=

    2025

  35. [43]

    Nature , volume=

    Towards multimodal foundation models in molecular cell biology , author=. Nature , volume=. 2025 , publisher=

    2025

  36. [44]

    European Conference on Computer Vision , pages=

    Sharegpt4v: Improving large multi-modal models with better captions , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  37. [45]

    International conference on machine learning , pages=

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  38. [46]

    Machine Vision and Applications , volume=

    Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition , author=. Machine Vision and Applications , volume=. 2021 , publisher=

    2021

  39. [47]

    Nature biomedical engineering , volume=

    A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics , author=. Nature biomedical engineering , volume=. 2023 , publisher=

    2023

  40. [48]

    Nature machine intelligence , volume=

    Multimodal data fusion for cancer biomarker discovery with deep learning , author=. Nature machine intelligence , volume=. 2023 , publisher=

    2023

  41. [49]

    ACM Computing Surveys , volume=

    Foundations & trends in multimodal machine learning: Principles, challenges, and open questions , author=. ACM Computing Surveys , volume=. 2024 , publisher=

    2024

  42. [50]

    CoRR , year=

    Re-Imagining Multimodal Instruction Tuning: A Representation View , author=. CoRR , year=

  43. [51]

    Advances in Neural Information Processing Systems , volume=

    Quantifying & modeling multimodal interactions: An information decomposition framework , author=. Advances in Neural Information Processing Systems , volume=

  44. [52]

    Multi30K: Multilingual English-German Image Descriptions

    Elliott, Desmond and Frank, Stella and Sima'an, Khalil and Specia, Lucia. Multi30K: Multilingual English-German Image Descriptions. Proceedings of the 5th Workshop on Vision and Language. 2016. doi:10.18653/v1/W16-3210

    2016 doi

  45. [53]

    Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Multimodal Transformer for Unaligned Multimodal Language Sequences , author=. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  46. [54]

    Information Fusion , volume=

    Triple disentangled representation learning for multimodal affective analysis , author=. Information Fusion , volume=. 2025 , publisher=

    2025

  47. [55]

    Advances in neural information processing systems , volume=

    Multibench: Multiscale benchmarks for multimodal representation learning , author=. Advances in neural information processing systems , volume=

  48. [56]

    Advances in Neural Information Processing Systems , volume=

    Asif: Coupled data turns unimodal models to multimodal without training , author=. Advances in Neural Information Processing Systems , volume=

  49. [57]

    Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) , pages=

    How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes , author=. Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) , pages=

  50. [58]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Lit: Zero-shot transfer with locked-image text tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  51. [59]

    Journal of Biomedical Informatics , pages=

    Evaluating an information theoretic approach for selecting multimodal data fusion methods , author=. Journal of Biomedical Informatics , pages=. 2025 , publisher=

    2025

  52. [60]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Multimodal representation learning by alternating unimodal adaptation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  53. [61]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Found in translation: Learning robust joint representations by cyclic translations between modalities , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  54. [62]

    Advances in Neural Information Processing Systems , volume=

    Factorized contrastive learning: Going beyond multi-view redundancy , author=. Advances in Neural Information Processing Systems , volume=

  55. [63]

    International Conference on Learning Representations , volume=

    Towards principled evaluations of sparse autoencoders for interpretability and control , author=. International Conference on Learning Representations , volume=

  56. [64]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Where and what? examining interpretable disentangled representations , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  57. [65]

    International Conference on Artificial Intelligence and Statistics , pages=

    Decoupling local and global representations of time series , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.