The reviewed record of science sign in
Pith

arxiv: 2412.15212 · v2 · pith:CE3YOYL2 · submitted 2024-12-19 · cs.CV · cs.AI· cs.LG

Scaling 4D Representations

Jo\~ao Carreira , Dilara Gokay , Michael King , Chuhan Zhang , Ignacio Rocco , Aravindh Mahendran , Thomas Albert Keck , Joseph Heyward
show 27 more authors
Skanda Koppula Etienne Pot Goker Erdogan Yana Hasson Yi Yang Klaus Greff Guillaume Le Moing Sjoerd van Steenkiste Daniel Zoran Drew A. Hudson Pedro V\'elez Luisa Polan\'ia Luke Friedman Chris Duvarney Ross Goroshin Kelsey Allen Jacob Walker Rishabh Kabra Eric Aboussouan Jennifer Sun Thomas Kipf Carl Doersch Viorica P\u{a}tr\u{a}ucean Dima Damen Pauline Luc Mehdi S. M. Sajjadi Andrew Zisserman
This is my paper

Reviewed by Pithpith:CE3YOYL2open to challenge →

classification cs.CV cs.AIcs.LG
keywords videolearningmodelsscalingself-supervisedtasksclassificationestimation
0
0 comments X
read the original abstract

Scaling has not yet been convincingly demonstrated for pure self-supervised learning from video. However, prior work has focused evaluations on semantic-related tasks $\unicode{x2013}$ action classification, ImageNet classification, etc. In this paper we focus on evaluating self-supervised learning on non-semantic vision tasks that are more spatial (3D) and temporal (+1D = 4D), such as camera pose estimation, point and object tracking, and depth estimation. We show that by learning from very large video datasets, masked auto-encoding (MAE) with transformer video models actually scales, consistently improving performance on these 4D tasks, as model size increases from 20M all the way to the largest by far reported self-supervised video model $\unicode{x2013}$ 22B parameters. Rigorous apples-to-apples comparison with many recent image and video models demonstrates the benefits of scaling 4D representations. Pretrained models are available at https://github.com/google-deepmind/representations4d .

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LA-Pose: Latent Action Pretraining Meets Pose Estimation

    cs.CV 2026-04 unverdicted novelty 6.0

    LA-Pose achieves over 10% higher pose accuracy than recent feed-forward methods on Waymo and PandaSet benchmarks by repurposing latent actions from self-supervised inverse-dynamics pretraining while using orders of ma...

  2. Frozen Forecasting: A Unified Evaluation

    cs.CV 2025-07 unverdicted novelty 6.0

    A new evaluation framework using latent diffusion on frozen vision backbones shows video-pretrained models consistently outperform image-based ones in forecasting entire trajectories across abstraction levels.

  3. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    cs.AI 2025-06 unverdicted novelty 6.0

    V-JEPA 2 pre-trained on massive unlabeled video achieves strong results on motion understanding and action anticipation, SOTA video QA at 8B scale, and enables zero-shot robotic planning on Franka arms using only 62 h...

  4. Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?

    cs.CV 2026-06 unverdicted novelty 5.0

    Systematic empirical comparison of temporal context placement across backbone, PEFT modules, and probes for low-resource video task adaptation on appearance, motion, and dense tasks.

  5. Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

    cs.CV 2026-05 unverdicted novelty 5.0

    Freezing an image foundation model and training only a recurrent temporal module yields strong temporal performance on video tasks without large-scale video pre-training.