pith. sign in

arxiv: 2607.01821 · v1 · pith:M55GSO55new · submitted 2026-07-02 · 📊 stat.ME

Pattern-Calibrated Multimodal Prediction under Blockwise Missingness

Junhan Yu , Kejian Zhang , Doudou Zhou , Guojun Zhu This is my paper

Pith reviewed 2026-07-03 08:16 UTC · model grok-4.3

classification 📊 stat.ME
keywords multimodal predictionblockwise missingnesspattern calibrationshared-specific alignmentnon-asymptotic boundsoverlap effective sample sizecalibration gap
0
0 comments X

The pith

MOSAIC borrows information across missingness patterns in multimodal data by fitting on overlaps then calibrating the gap with target-pattern samples.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper treats blockwise missingness as a problem of distinct observed-modality patterns rather than incomplete inputs. It introduces MOSAIC, which learns shared and modality-specific representations, fits a first-stage predictor on the available overlaps, and then adjusts for the difference between that predictor and the target pattern using only target data. Non-asymptotic error bounds decompose the excess risk into three explicit terms: effective sample size from overlaps, size of the calibration gap, and representation-learning error. These bounds identify the regimes where borrowing across patterns beats purely local fitting and the regimes where rule mismatch or poor representations erase the gain. Simulations and three real applications (ICU mortality, emotion recognition, glaucoma) illustrate the decomposition in practice.

Core claim

By separating overlap-based representation learning and first-stage prediction from a subsequent calibration step estimated solely on target-pattern observations, the procedure yields predictors whose error is controlled by the product of overlap sample size and calibration gap plus representation error, rather than by the full pooled sample or by imputation artifacts.

What carries the argument

The MOSAIC procedure: overlap-aware shared-specific alignment followed by inter-pattern calibration that estimates the additive gap from target data alone.

If this is right

  • When the calibration gap is small relative to overlap sample size, borrowing improves finite-sample risk over fitting only on the target pattern.
  • Representation-learning error appears as an additive term that cannot be reduced by more target data alone.
  • Rule mismatch between patterns acts as a hard limit on the benefit of borrowing, independent of sample sizes.
  • The same decomposition quantifies the cost of using a single pooled model versus pattern-specific calibration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The framework suggests collecting a modest calibration set for each clinically important pattern rather than maximizing total sample size across all patterns.
  • It may generalize to other structured missingness where the observed set defines a distinct conditional distribution.
  • In deployment, one could monitor the estimated calibration gap on incoming target data to decide whether to switch from the borrowed predictor to a purely local one.

Load-bearing premise

Target-pattern observations are numerous enough to estimate the calibration gap reliably and the first-stage predictor on overlaps introduces no bias that calibration cannot remove.

What would settle it

An experiment in which the measured calibration gap fails to shrink with additional target samples or in which the total error exceeds the local-only error even when the estimated gap is near zero.

Figures

Figures reproduced from arXiv: 2607.01821 by Doudou Zhou, Guojun Zhu, Junhan Yu, Kejian Zhang.

Figure 1
Figure 1. Figure 1: Blockwise missingness induces pattern-specific prediction tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representation recovery in the Experiment 1 sample-size sweep. Shaded bands [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representation recovery as q varies, with N = 5000. Shaded bands are pointwise 95% Monte Carlo confidence intervals for the mean over 100 replications. separate the contributions of overlap borrowing, target-pattern calibration, pooled fitting, and representation learning. We maintain L = 3 modalities, and the covariate generation settings for the linear and nonlinear regimes follow Experiment 1. The respo… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison between MOSAIC and external baselines in Experiment 2. Shaded [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Downstream decomposition as residual dependence among modality-specific [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overall and missingness-pattern-specific performance on MIMIC next-visit ICU [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Component analysis of MOSAIC on MIMIC next-visit ICU mortality prediction. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
read the original abstract

Blockwise missingness in multimodal data is usually treated as an incomplete-input problem. We instead focus on prediction for a prespecified observed-modality pattern, where the observed modality set determines the information on which the prediction rule can condition. A procedure that imputes missing modalities, zero-fills unobserved modalities, or trains a single pooled predictor may borrow information across patterns, but it can also mix pattern-specific prediction rules. We propose Multimodal Overlap-aware Shared-specific Alignment and Inter-pattern Calibration (MOSAIC), a pattern-calibrated framework for borrowing across missingness patterns without collapsing their prediction rules. MOSAIC learns shared and modality-specific representations, uses the available representations that overlap with the target pattern to fit a first-stage predictor, and then estimates the calibration gap from target-pattern data. We establish non-asymptotic bounds that decompose the error into overlap effective sample size, calibration gap, and representation-learning error, clarifying when cross-pattern borrowing improves over local fitting and when the improvement is controlled by rule mismatch or representation-learning error. Simulations examine representation recovery and target-pattern correction, and applications to ICU mortality prediction, emotion recognition, and glaucoma classification show gains when target-pattern samples are limited or pattern-specific rules differ.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript proposes MOSAIC (Multimodal Overlap-aware Shared-specific Alignment and Inter-pattern Calibration), a pattern-calibrated framework for multimodal prediction under blockwise missingness that targets a prespecified observed-modality pattern. It learns shared and modality-specific representations, fits a first-stage predictor on representations overlapping the target pattern, and estimates a calibration gap from target-pattern data. Non-asymptotic bounds are derived that decompose prediction error into overlap effective sample size, calibration gap, and representation-learning error, with the goal of clarifying when cross-pattern borrowing improves over local fitting. Simulations study representation recovery and target correction; applications to ICU mortality, emotion recognition, and glaucoma classification report gains when target samples are limited or pattern-specific rules differ.

Significance. If the claimed error decomposition is valid and the procedure avoids irreducible bias from the first-stage predictor, the work supplies a theoretically grounded approach to borrowing across missingness patterns without collapsing distinct prediction rules. This could be valuable in domains with structured multimodal missingness, such as clinical prediction, where target-pattern data are scarce.

major comments (1)
  1. [Abstract] Abstract: the non-asymptotic bounds are asserted as decomposing error into overlap effective sample size, calibration gap, and representation-learning error, but no derivation steps, explicit assumptions on the representation-learning procedure, or conditions ensuring the calibration gap is estimable without irreducible bias are visible; this is load-bearing for the central claim that the bounds clarify when borrowing helps versus when it is controlled by rule mismatch or representation error.
minor comments (1)
  1. [Abstract] The abstract introduces the MOSAIC acronym and procedure but does not define the precise form of the first-stage predictor or how the calibration gap is estimated from target data; adding one sentence on these operational details would improve readability.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful reading of the manuscript and for identifying this point about the abstract. We respond to the major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the non-asymptotic bounds are asserted as decomposing error into overlap effective sample size, calibration gap, and representation-learning error, but no derivation steps, explicit assumptions on the representation-learning procedure, or conditions ensuring the calibration gap is estimable without irreducible bias are visible; this is load-bearing for the central claim that the bounds clarify when borrowing helps versus when it is controlled by rule mismatch or representation error.

    Authors: The abstract is a concise high-level summary and does not contain full derivations or assumption statements, which is standard due to length constraints. The non-asymptotic bounds and their decomposition into overlap effective sample size, calibration gap, and representation-learning error are derived in Section 4, with complete proof steps in Appendix A. Explicit assumptions on the representation-learning procedure (including bounded representation error and consistency of overlap-aware alignment) appear as Assumptions 1-3 in Section 3.1. Conditions ensuring the calibration gap is estimable without irreducible bias from the first-stage predictor are stated in Theorem 2, which requires positive target-pattern sample size and controlled representation error so that the gap term vanishes with the representation error. These results support the central claim by separating the error sources, enabling analysis of when cross-pattern borrowing reduces error relative to local fitting. We do not view changes to the abstract as necessary. revision: no

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The paper defines the MOSAIC procedure (shared/specific representations, first-stage predictor on overlaps, target-pattern calibration gap estimation) and derives non-asymptotic error bounds that decompose into overlap effective sample size, calibration gap, and representation-learning error. These quantities are constructed directly from the observed data and procedure components; the bounds clarify when borrowing improves over local fitting without any step reducing the claimed improvement to a fitted parameter already present in the target pattern or to a self-citation chain. The derivation is therefore self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the method description implies standard representation-learning assumptions and the existence of a calibration gap, but none are enumerated.

pith-pipeline@v0.9.1-grok · 5745 in / 1178 out tokens · 21535 ms · 2026-07-03T08:16:24.665339+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

151 extracted references · 23 canonical work pages · 9 internal anchors

  1. [1]

    A General Representation-Based Approach to Multi-Source Domain Adaptation

    A General Representation-Based Approach to Multi-Source Domain Adaptation , author=. arXiv preprint arXiv:2604.23790 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Proceedings of the 29th ACM International Conference on Multimedia , pages =

    Transformer-Based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis , author =. Proceedings of the 29th ACM International Conference on Multimedia , pages =. 2021 , doi =

    2021

  3. [3]

    Molecular Systems Biology , volume =

    Multi-Omics Factor Analysis---a framework for unsupervised integration of multi-omics data sets , author =. Molecular Systems Biology , volume =. 2018 , doi =

    2018

  4. [4]

    Biometrika , volume =

    Inference and Missing Data , author =. Biometrika , volume =. 1976 , doi =

    1976

  5. [5]

    2019 , doi =

    Statistical Analysis with Missing Data , author =. 2019 , doi =

    2019

  6. [6]

    2026 , issn =

    Multimodal fusion on low-quality data: A comprehensive survey , journal =. 2026 , issn =. doi:https://doi.org/10.1016/j.inffus.2026.104437 , author =

    work page doi:10.1016/j.inffus.2026.104437 2026

  7. [7]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Missing Modalities Imputation via Cascaded Residual Autoencoder , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  8. [8]

    IEEE Transactions on Medical Imaging , volume =

    Multimodal MR Synthesis via Modality-Invariant Latent Representation , author =. IEEE Transactions on Medical Imaging , volume =. 2018 , doi =

    2018

  9. [9]

    Advances in Neural Information Processing Systems , volume =

    Multimodal Generative Models for Scalable Weakly-Supervised Learning , author =. Advances in Neural Information Processing Systems , volume =

  10. [10]

    Advances in Neural Information Processing Systems , volume =

    Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models , author =. Advances in Neural Information Processing Systems , volume =

  11. [11]

    and Hoadley, Katherine A

    Lock, Eric F. and Hoadley, Katherine A. and Marron, J. S. and Nobel, Andrew B. , journal =. Joint and Individual Variation Explained (. 2013 , doi =

    2013

  12. [12]

    Journal of Multivariate Analysis , volume =

    Angle-Based Joint and Individual Variation Explained , author =. Journal of Multivariate Analysis , volume =. 2018 , doi =

    2018

  13. [13]

    Biometrics , volume =

    Structural Learning and Integrative Decomposition of Multi-View Data , author =. Biometrics , volume =. 2019 , doi =

    2019

  14. [14]

    Biometrika , volume =

    A Spectral Framework for Multi-View Subspace Learning Using the Product of Projections , author =. Biometrika , volume =. 2026 , doi =

    2026

  15. [15]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , volume =

    Transfer Learning for High-Dimensional Linear Regression , author =. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume =. 2022 , doi =

    2022

  16. [16]

    Journal of the American Statistical Association , volume =

    Transfer Learning Under High-Dimensional Generalized Linear Models , author =. Journal of the American Statistical Association , volume =. 2023 , doi =

    2023

  17. [17]

    Information Fusion , volume =

    Review of multimodal machine learning approaches in healthcare , author =. Information Fusion , volume =. 2025 , doi =

    2025

  18. [18]

    Transactions on Machine Learning Research , issn=

    Deep Multimodal Learning with Missing Modality: A Survey , author=. Transactions on Machine Learning Research , issn=. 2026 , note=

    2026

  19. [19]

    , journal=

    Zhang, Yue and Peng, Chengtao and Wang, Qiuli and Song, Dan and Li, Kaiyan and Kevin Zhou, S. , journal=. Unified Multi-Modal Image Synthesis for Missing Modality Imputation , year=

  20. [20]

    Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities , author =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages =. 2021 , address =

    2021

  21. [21]

    ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages =

    Exploiting Modality-Invariant Feature for Robust Multimodal Emotion Recognition with Missing Modalities , author =. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages =. 2023 , doi =

    2023

  22. [22]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Multi-Modal Learning With Missing Modality via Shared-Specific Feature Modelling , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  23. [23]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    MMANet: Margin-Aware Distillation and Modality-Aware Regularization for Incomplete Multimodal Learning , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  24. [24]

    Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis , DOI =

    Yuan, Ziqi and Li, Wei and Xu, Hua and Yu, Wenmeng , year =. Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis , DOI =. Proceedings of the 29th ACM International Conference on Multimedia , publisher =

  25. [25]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =. 2024 , doi =

    2024

  26. [26]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =

    Multimodal Machine Learning: A Survey and Taxonomy , author =. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =. 2019 , doi =

    2019

  27. [27]

    Proceedings of the 28th ACM International Conference on Multimedia , pages =

    MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis , author =. Proceedings of the 28th ACM International Conference on Multimedia , pages =. 2020 , doi =

    2020

  28. [28]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages =

    Private-Shared Disentangled Multimodal VAE for Learning of Latent Representations , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages =. 2021 , doi =

    2021

  29. [29]

    IEEE Transactions on Knowledge and Data Engineering , volume =

    A Survey on Transfer Learning , author =. IEEE Transactions on Knowledge and Data Engineering , volume =. 2010 , doi =

    2010

  30. [30]

    Advances in Neural Information Processing Systems 21 , pages =

    Domain Adaptation with Multiple Sources , author =. Advances in Neural Information Processing Systems 21 , pages =

  31. [31]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Multi-source domain adaptation: A causal view , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  32. [32]

    Journal of Machine Learning Research , volume =

    Invariant Models for Causal Transfer Learning , author =. Journal of Machine Learning Research , volume =

  33. [33]

    Attention Bottlenecks for Multimodal Fusion , url =

    Nagrani, Arsha and Yang, Shan and Arnab, Anurag and Jansen, Aren and Schmid, Cordelia and Sun, Chen , booktitle =. Attention Bottlenecks for Multimodal Fusion , url =

  34. [34]

    Machine Learning for Healthcare Conference , pages=

    Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention , author=. Machine Learning for Healthcare Conference , pages=. 2023 , organization=

    2023

  35. [35]

    The Twelfth International Conference on Learning Representations , year=

    Multimodal patient representation learning with missing modalities and labels , author=. The Twelfth International Conference on Learning Representations , year=

  36. [36]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Multi-modal learning with missing modality via shared-specific feature modelling , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  37. [37]

    Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages=

    M3care: Learning with missing modalities in multimodal healthcare data , author=. Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages=

  38. [38]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  39. [39]

    Representation Learning with Contrastive Predictive Coding

    Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  40. [40]

    2024 , month = mar, note =

    Johnson, Alistair and Lungren, Matthew and Peng, Yifan and Lu, Zhiyong and Mark, Roger and Berkowitz, Seth and Horng, Steven , title =. 2024 , month = mar, note =

    2024

  41. [41]

    Nature communications , volume=

    The medical segmentation decathlon , author=. Nature communications , volume=. 2022 , publisher=

    2022

  42. [42]

    2023 , publisher=

    Johnson, Alistair EW and Bulgarelli, Lucas and Shen, Lu and Gayles, Alvin and Shammout, Ayad and Horng, Steven and Pollard, Tom J and Hao, Sicheng and Moody, Benjamin and Gow, Brian and others , journal=. 2023 , publisher=

    2023

  43. [43]

    Scientific data , volume=

    The eICU Collaborative Research Database, a freely available multi-center database for critical care research , author=. Scientific data , volume=. 2018 , publisher=

    2018

  44. [44]

    Computer Vision--ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2--6, 2018, Revised Selected Papers, Part V 14 , pages=

    VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video , author=. Computer Vision--ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2--6, 2018, Revised Selected Papers, Part V 14 , pages=. 2019 , organization=

    2018

  45. [45]

    npj Digital Medicine , volume=

    Multimodal machine learning in precision health: A scoping review , author=. npj Digital Medicine , volume=. 2022 , publisher=

    2022

  46. [46]

    AMIA Annual Symposium Proceedings , volume=

    A multimodal transformer: Fusing clinical notes with structured ehr data for interpretable in-hospital mortality prediction , author=. AMIA Annual Symposium Proceedings , volume=

  47. [47]

    International conference on machine learning , pages=

    The effect of natural distribution shift on question answering models , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  48. [48]

    International conference on machine learning , pages=

    Do imagenet classifiers generalize to imagenet? , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  49. [49]

    Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection , url =

    Park, Chunjong and Awadalla, Anas and Kohno, Tadayoshi and Patel, Shwetak , booktitle =. Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection , url =

  50. [50]

    Split learning for health: Distributed deep learning without sharing raw patient data

    Split learning for health: Distributed deep learning without sharing raw patient data , author=. arXiv preprint arXiv:1812.00564 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  51. [51]

    Scientific reports , volume=

    Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data , author=. Scientific reports , volume=. 2020 , publisher=

    2020

  52. [52]

    Le and Thu Nguyen and Michael A

    Lien P. Le and Thu Nguyen and Michael A. Riegler and Pål Halvorsen and Binh T. Nguyen , keywords =. Multimodal missing data in healthcare: A comprehensive review and future directions , journal =. 2025 , issn =. doi:https://doi.org/10.1016/j.cosrev.2024.100720 , url =

    work page doi:10.1016/j.cosrev.2024.100720 2025

  53. [53]

    , author=

    Multimodal deep learning. , author=. ICML , volume=

  54. [54]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Smil: Multimodal learning with severely missing modality , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  55. [55]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Are multimodal transformers robust to missing modality? , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  56. [56]

    and Ma, Jieming and Cui, Zhiming , TITLE =

    Xu, Jiabao and Xi, Xuefeng and Chen, Jie and Sheng, Victor S. and Ma, Jieming and Cui, Zhiming , TITLE =. Applied Sciences , VOLUME =. 2022 , NUMBER =

    2022

  57. [57]

    Machine Learning for Healthcare Conference , pages=

    MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images , author=. Machine Learning for Healthcare Conference , pages=. 2022 , organization=

    2022

  58. [58]

    Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages=

    Hgmf: heterogeneous graph-based fusion for multimodal data with incompleteness , author=. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages=

  59. [59]

    Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages=

    Multimodal learning with incomplete modalities by knowledge distillation , author=. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages=

  60. [60]

    Machine Learning for Health (ML4H) , pages=

    Multimodal pretraining of medical time series and notes , author=. Machine Learning for Health (ML4H) , pages=. 2023 , organization=

    2023

  61. [61]

    Computers in Biology and Medicine , volume=

    Multimodal image encoding pre-training for diabetic retinopathy grading , author=. Computers in Biology and Medicine , volume=. 2022 , publisher=

    2022

  62. [62]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Best of both worlds: Multimodal contrastive learning with tabular and imaging data , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  63. [63]

    IEEE journal of biomedical and health informatics , volume=

    Hi-BEHRT: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records , author=. IEEE journal of biomedical and health informatics , volume=. 2022 , publisher=

    2022

  64. [64]

    Proceedings of the Conference on Empirical Methods in Natural Language Processing

    Hierarchical pretraining on multimodal electronic health records , author=. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing , volume=

  65. [65]

    IEEE Journal of Biomedical and Health Informatics , volume=

    Multimodal data matters: Language model pre-training over structured and unstructured electronic health records , author=. IEEE Journal of Biomedical and Health Informatics , volume=. 2022 , publisher=

    2022

  66. [66]

    IEEE journal of biomedical and health informatics , volume=

    Bidirectional representation learning from transformers using multimodal electronic health record data to predict depression , author=. IEEE journal of biomedical and health informatics , volume=. 2021 , publisher=

    2021

  67. [67]

    International conference on machine learning , pages=

    Understanding contrastive representation learning through alignment and uniformity on the hypersphere , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  68. [68]

    and Lee, Sungbok and Narayanan, Shrikanth S

    Busso, Carlos and Bulut, Murtaza and Lee, Chi-Chun and Kazemzadeh, Abe and Mower, Emily and Kim, Samuel and Chang, Jeannette N. and Lee, Sungbok and Narayanan, Shrikanth S. , year =. IEMOCAP: interactive emotional dyadic motion capture database , volume =. Language Resources and Evaluation , publisher =. doi:10.1007/s10579-008-9076-6 , number =

    work page doi:10.1007/s10579-008-9076-6

  69. [69]

    FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification , booktitle =

    Tian, Yu and Wen, Congcong and Shi, Min and Afzal, Muhammad Muneeb and Huang, Hao and Khan, Muhammad Osama and Luo, Yan and Fang, Yi and Wang, Mengyu , editor =. FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification , booktitle =. 2025 , publisher =. doi:10.1007/978-3-031-73116-7_15 , isbn =

    work page doi:10.1007/978-3-031-73116-7_15 2025

  70. [70]

    IEEE Journal of Selected Topics in Signal Processing , volume=

    WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing , author=. IEEE Journal of Selected Topics in Signal Processing , volume=. 2022 , publisher=

    2022

  71. [71]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , pages=

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , pages=

    2019

  72. [72]

    MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers , volume =

    Wang, Wenhui and Wei, Furu and Dong, Li and Bao, Hangbo and Yang, Nan and Zhou, Ming , booktitle =. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers , volume =

  73. [73]

    Biostatistics , volume =

    Fletcher Mercaldo, Sarah and Blume, Jeffrey D , title =. Biostatistics , volume =. 2020 , month =. doi:10.1093/biostatistics/kxy040 , eprint =

    work page doi:10.1093/biostatistics/kxy040 2020

  74. [74]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Sharing Pattern Submodels for Prediction with Missing Values , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2023 , doi=

    2023

  75. [75]

    2018 , publisher=

    High-dimensional probability: An introduction with applications in data science , author=. 2018 , publisher=

    2018

  76. [76]

    Deep Multimodal Representation Learning: A Survey , year=

    Guo, Wenzhong and Wang, Jianwen and Wang, Shiping , journal=. Deep Multimodal Representation Learning: A Survey , year=

  77. [77]

    DeepSeek-VL: Towards Real-World Vision-Language Understanding

    Deepseek-vl: towards real-world vision-language understanding , author=. arXiv preprint arXiv:2403.05525 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  78. [78]

    International conference on machine learning , pages=

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  79. [79]

    ACM Computing Surveys , volume=

    Multimodal sentiment analysis: a survey of methods, trends, and challenges , author=. ACM Computing Surveys , volume=. 2023 , publisher=

    2023

  80. [80]

    Information Fusion , volume=

    Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions , author=. Information Fusion , volume=. 2023 , publisher=

    2023

Showing first 80 references.