REVIEW 6 cited by
UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Learning discriminative spatiotemporal representation is the key problem of video understanding. Recently, Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. Unfortunately, they exhibit limitations in tackling local video redundancy, due to the blind global comparison among tokens. UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format. However, this model has to require a tiresome and complicated image-pretraining phrase, before being finetuned on videos. This blocks its wide usage in practice. On the contrary, open-sourced ViTs are readily available and well-pretrained with rich image supervision. Based on these observations, we propose a generic paradigm to build a powerful family of video networks, by arming the pretrained ViTs with efficient UniFormer designs. We call this family UniFormerV2, since it inherits the concise style of the UniFormer block. But it contains brand-new local and global relation aggregators, which allow for preferable accuracy-computation balance by seamlessly integrating advantages from both ViTs and UniFormer. Without any bells and whistles, our UniFormerV2 gets the state-of-the-art recognition performance on 8 popular video benchmarks, including scene-related Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. Code will be available at https://github.com/OpenGVLab/UniFormerV2.
Forward citations
Cited by 6 Pith papers
-
Decoding Children's Gait Behavior
A new 1,185-video pediatric gait dataset with EVGS labels, plus a VideoMAE-based model, reaches ~84% average accuracy on 34 clinical gait items — far above MLLMs and prior gait models.
-
PhiZero: A World Model Built Around Physical Language
A self-supervised discrete physical-language bottleneck plus a VLM reasoner lets a world model predict state transitions before rendering video, improving physical coherence and enabling zero-shot motion transfer.
-
Rethinking the Readout: Unlocking Video Backbones for AI-Generated Video Detection
Replacing the global-pooling readout of a frozen video backbone with a velocity-gated, per-channel-magnitude readout improves AI-generated video detection cross-generator accuracy by several AUC points.
-
IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes
A new multi-shot video dataset and an instance-prompt video LLM report large gains, but the main benchmark is built by the same authors and the model is not released.
-
OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning
A shared-backbone transformer with pairwise modality training reports top results across 25 datasets spanning 12 modalities.
-
Uncertainty-aware Diffusion and Reinforcement Learning for Joint Plane Localization and Anomaly Diagnosis in 3D Ultrasound
A diffusion-plus-reinforcement-learning system jointly localizes the coronal plane and diagnoses congenital uterine anomalies in 3D ultrasound, with an uncertainty adjustment reaching 94.58% classification accuracy an...
Discussion (0). Sign in to comment.