REVIEW 6 cited by
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.
Forward citations
Cited by 6 Pith papers
-
CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets
CM3T shows that multi-head vision adapters plus cross-attention adapters can adapt frozen supervised-pretrained video transformers with a fraction of the trainable parameters of full fine-tuning.
-
RPN 2: On Interdependence Function Learning Towards Unifying and Advancing CNN, RNN, GNN, and Transformer
RPN 2 adds interdependence functions to the Reconciled Polynomial Network and claims that CNN, RNN, GNN, and Transformer differ only in which interdependence function they use.
-
SubstationAI: Multimodal Large Model-Based Approaches for Analyzing Substation Equipment Faults
SubstationAI, a fine-tuned LLaVA-1.5-7B model augmented with a fault knowledge base, receives higher expert ratings than GPT-4 for substation fault reports, but suspected train/test overlap makes the result unreliable.
-
CrossVIT-augmented Geospatial-Intelligence Visualization System for Tracking Economic Development Dynamics
A multimodal satellite and street view deep learning system predicts nighttime-light-derived proxy scores for Chinese counties with R-squared 0.8363 and visualizes them on a web map.
-
A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning
A review that categorizes deep audio-visual correlation learning methods by architectures, objective functions, datasets, and evaluation metrics, and points to missing standardized benchmarks.
-
Federated Learning Inspired Fuzzy Systems: Decentralized Rule Updating for Privacy and Scalable Decision Making
The paper suggests federated-learning-style updates for fuzzy rule sets and a machine-learning-augmented fuzzy system, without providing implementation, derivation, or evidence.
Discussion (0). Continue with ORCID to comment.