REVIEW 7 cited by
Audiovisual SlowFast Networks for Video Recognition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, we perform hierarchical audiovisual synchronization to learn joint audiovisual features. We report state-of-the-art results on six video action classification and detection datasets, perform detailed ablation studies, and show the generalization of AVSlowFast to learn self-supervised audiovisual features. Code will be made available at: https://github.com/facebookresearch/SlowFast.
Forward citations
Cited by 7 Pith papers
-
EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models
EPFL-Smart-Kitchen-30 is a 29.7-hour multimodal cooking dataset with 60k action segments and four benchmarks, including a kinematic-focused VQA benchmark that shows current video-language models struggle with hand and...
-
Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos
LG-CAV-MAE trains a tri-modal audio-visual-text model on CLAP-filtered auto-generated captions and beats existing audio-visual masked autoencoders on retrieval and classification.
-
DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation
DMAF-Net improves incomplete multi-modal MRI segmentation by dynamically masking missing modalities, aligning uni-modal and fused features through covariance, attention, and prototype losses, and reweighting training ...
-
TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
Weakly supervised vision-language model jointly generating open-ended video QA answers with temporal groundings, reporting SOTA on NExT-GQA, MSVD-QA, and ActivityNet-QA.
-
Attention-Driven Multimodal Alignment for Long-term Action Quality Assessment
LMAC-Net reports state-of-the-art Spearman correlations on the RG and Fis-V benchmarks by aligning attention centers across RGB, optical flow, and audio branches.
-
OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning
A shared-backbone transformer with pairwise modality training reports top results across 25 datasets spanning 12 modalities.
-
DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding
DEL is a new audio-visual transformer framework that reports state-of-the-art temporal action localization on UnAV-100, THUMOS14, ActivityNet 1.3, and EPIC-Kitchens-100.
Discussion (0). Sign in to comment.