REVIEW 9 cited by
VideoBERT: A Joint Model for Video and Language Representation Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.
Forward citations
Cited by 9 Pith papers
-
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
A two-stream BERT-style model pretrained on weakly aligned image-caption data transfers to VQA, VCR, referring expressions, and retrieval, outperforming task-specific models on all four.
-
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
LXMERT pretrains a three-encoder Transformer on image-sentence pairs with five tasks and achieves state-of-the-art VQA, GQA, and NLVR2 results after fine-tuning.
-
Fusion of Detected Objects in Text for Visual Question Answering
Injecting detected object features into BERT's token embeddings (early fusion) improved visual question answering and achieved state-of-the-art results on VCR.
-
CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models
CLIP-CC-Bench is a 200-clip benchmark with expert paragraph references that ranks 17 video-language models via an ensemble of five embedding-based semantic judges.
-
AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction
AdaCM2 adaptively evicts low-attention video tokens using cross-modality attention scores, keeping memory bounded while improving accuracy on long-video QA, captioning, and classification.
-
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
VL-BERT pre-trains a single-stream Transformer on image captions and text, and the resulting representation improves VCR, VQA, and RefCOCO+ benchmarks.
-
Integrating Multimodal Information in Large Pretrained Transformers
The Multimodal Adaptation Gate lets BERT and XLNet be fine-tuned with visual and acoustic inputs, improving multimodal sentiment analysis and reaching human-level scores on CMU-MOSI.
-
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
Image-caption pre-training with masked language, masked object, and image-text matching objectives improves image-text retrieval and, to a lesser extent, visual commonsense reasoning.
-
The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
A survey tracing the evolution of visual question answering from 2015 CNN-LSTM models through attention mechanisms, modular networks, vision-language pretraining, and large multimodal models.
Discussion (0). Continue with ORCID to comment.