REVIEW 6 cited by
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and pre-trained models are available at https://github.com/salesforce/ALBEF/.
Forward citations
Cited by 6 Pith papers
-
A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysis
Fusing labs, ECG, and echo through a text-guided transformer beats single-modality models for heart failure diagnosis (AUC 0.91) and incident-HF risk (C-index 0.61) on a newly curated MIMIC-IV cohort.
-
Multi-modal encoder-decoder neural network for forecasting solar wind speed at L1
This paper presents a multi-modal encoder-decoder network that forecasts daily averaged solar wind speed at L1 up to four days ahead, with validation RMSE around 55 to 58 km/s and a four-day-ahead test RMSE of 53 km/s.
-
Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translation
Pre-trained decoders consistently improve multimodal translation, while pre-trained encoders help only when visual-text alignment is strong.
-
MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence
MEDIC-AD adds anomaly-aware and difference tokens to a medical VLM, claiming SOTA lesion detection, temporal tracking, and visual grounding; the zero-shot claim is undermined by likely train/test overlap.
-
Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
CoCoA forces an MLLM to reconstruct masked text through a single EOS token, improving multimodal embedding quality on MMEB-V1 and matching MoCa at 3B with far less pretraining data.
-
Advancing Stroke Risk Prediction Using a Multi-modal Foundation Model
A CLIP-style multimodal model with image-tabular matching achieves modest AUC gains over unimodal baselines for pre-stroke stroke risk prediction on a small UK Biobank test set.
Discussion (0). Continue with ORCID to comment.