A boundary-forcing masked modeling paradigm for self-supervised vision pretraining yields a 1B model rivaling 7B models on dense spatial perception tasks.
hub Mixed citations
V-JEPA 2.1: Unlocking dense features in video self-supervised learning
Mixed citation behavior. Most common role is background (57%).
37
Pith papers citing it
Background
57% of classified citations
abstract
We present V-JEPA 2.1, a family of self-supervised models that learn dense, high-quality visual representations for both images and videos while retaining strong global scene understanding. The approach combines four key components. First, a dense predictive loss uses a masking-based objective in which both visible and masked tokens contribute to the training signal, encouraging explicit spatial and temporal grounding. Second, deep self-supervision applies the self-supervised objective hierarchically across multiple intermediate encoder layers to improve representation quality. Third, multi-modal tokenizers enable unified training across images and videos. Finally, the model benefits from effective scaling in both model capacity and training data. Together, these design choices produce representations that are spatially structured, semantically coherent, and temporally consistent. Empirically, V-JEPA 2.1 achieves state-of-the-art performance on several challenging benchmarks, including 7.71 mAP on Ego4D for short-term object-interaction anticipation and 40.8 Recall@5 on EPIC-KITCHENS for high-level action anticipation, as well as a 20-point improvement in real-robot grasping success rate over V-JEPA-2 AC. The model also demonstrates strong performance in robotic navigation (5.687 ATE on TartanDrive), depth estimation (0.307 RMSE on NYUv2 with a linear probe), and global recognition (77.7 on Something-Something-V2). These results show that V-JEPA 2.1 significantly advances the state of the art in dense visual understanding and world modeling.
hub tools
citation-role summary
background 4
method 3
citation-polarity summary
years
2026 37representative citing papers
GEAR jointly trains VQ tokenizer and AR generator end-to-end via dual hard/soft read-out and representation alignment, achieving up to 10x faster ImageNet gFID convergence than LlamaGen-REPA while generalizing across quantizers and to text-to-image.
OctoSense supplies a large multimodal robotics dataset and a late-fusion masked autoencoder that runs fast and outperforms image-only models on optical flow, depth, segmentation, and ego-motion tasks while remaining robust under sensor degradation.
USS is an end-to-end framework for embodied visual tracking that fuses text, point, box, and mask prompts via modality-specific encoders and hybrid attention, augmented by a latent world model, and demonstrates higher success rates with spatial cues on real robots and competitive simulation performa
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
RLA-WM predicts residual latent actions via flow matching to create visual feature world models that outperform prior feature-based and diffusion approaches while enabling offline video-based robot RL.
LookWhen factorizes video recognition into learning when, where, and what to compute via uniqueness-based token selection and dual-teacher distillation, achieving better accuracy-FLOPs trade-offs than baselines on multiple datasets.
World models succeed when their latent states are built to meet task-specific sufficiency constraints rather than preserving the maximum amount of information.
Being-H0.7 adds future-aware latent reasoning to direct VLA policies via dual-branch alignment on latent queries, matching world-model benefits at VLA efficiency.
Frozen video diffusion models, probed at optimal depth and noise levels, produce representations competitive with discriminative encoders across semantic and geometric video tasks in a single forward pass.
A self-supervised framework learns implicit 3D physics by lifting V-JEPA features into voxels and performing volumetric feature advection conditioned on actions.
P-JEPA enables long-form procedural video understanding by predicting pooled masked latent vectors in a dense frame-aligned action space, achieving SOTA fine-grained action classification on EgoExo4D with an order of magnitude fewer parameters than LLM methods.
Active inference supplies a test-time scaling law for physical AI agents by framing policy updates as soft Bayesian inference that reduces expected prediction errors, with a variational solution shown to outperform Q-learning and Bayesian RL in driving simulations.
GLINT introduces sparsely gated alignment and dense feature regularization on top of DINOv3 and V-JEPA encoders to enable query-specific zero-shot grounding and segmentation in 2D CXR and 3D CT.
Empirical study introduces behavioral and representational diagnostics showing architecture-dependent gains in object targeting and predictive structure for WAMs over VLAs on LIBERO and RoboTwin2.0.
TrajPilot predicts candidate future trajectories from egocentric context and uses them to condition action prediction in an embedding space, outperforming VLM and planner baselines on Ego-Exo4D, Ego4D, and other datasets with gains increasing at longer horizons.
Latent prediction video models exhibit a distinct robustness profile across corruption, occlusion, fine-grained discrimination, and temporal sensitivity compared to other self-supervised video models when used as world models.
Method converts exocentric videos to egocentric format via body-pose extraction and kinematics to improve egocentric world-model prediction and planning.
SARA introduces semantic saliency to guide relational alignment in video diffusion models, improving text following and motion quality over prior alignment methods.
Converting 3D MRI volumes into action-conditioned 2D slice navigation sequences offers a complementary self-supervised pretraining signal for learning anatomical and spatial representations.
Planning with a frozen egocentric world model is lifted from 48-dim joint actions to a few 2D goal waypoints via a trained policy, reducing CEM error reduction by 3.8x.
Einstein World Models integrate visual rollouts from a callable world-module into LLM reasoning traces to support complex thought beyond language.
Kairos learns and maintains control-sufficient world states via a cross-embodiment curriculum, hybrid linear temporal attention, and deployment-aware co-design for Physical AI.
The paper proposes an L0-L7 evidential ladder for evaluating world models in embodied decision-making, prioritizing interventional action fidelity and policy optimization utility over visual plausibility.
citing papers explorer
-
Vision Pretraining for Dense Spatial Perception
A boundary-forcing masked modeling paradigm for self-supervised vision pretraining yields a 1B model rivaling 7B models on dense spatial perception tasks.
-
GEAR: Guided End-to-End AutoRegression for Image Synthesis
GEAR jointly trains VQ tokenizer and AR generator end-to-end via dual hard/soft read-out and representation alignment, achieving up to 10x faster ImageNet gFID convergence than LlamaGen-REPA while generalizing across quantizers and to text-to-image.
-
OctoSense: Self-Supervised Learning for Multimodal Robot Perception
OctoSense supplies a large multimodal robotics dataset and a late-fusion masked autoencoder that runs fast and outperforms image-only models on optical flow, depth, segmentation, and ego-motion tasks while remaining robust under sensor degradation.
-
USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning
USS is an end-to-end framework for embodied visual tracking that fuses text, point, box, and mask prompts via modality-specific encoders and hybrid attention, augmented by a latent world model, and demonstrates higher success rates with spatial cues on real robots and competitive simulation performa
-
Learn from your own latents and not from tokens: A sample-complexity theory
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
-
Learning Visual Feature-Based World Models via Residual Latent Action
RLA-WM predicts residual latent actions via flow matching to create visual feature world models that outperform prior feature-based and diffusion approaches while enabling offline video-based robot RL.
-
LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute
LookWhen factorizes video recognition into learning when, where, and what to compute via uniqueness-based token selection and dual-teacher distillation, achieving better accuracy-FLOPs trade-offs than baselines on multiple datasets.
-
Latent State Design for World Models under Sufficiency Constraints
World models succeed when their latent states are built to meet task-specific sufficiency constraints rather than preserving the maximum amount of information.
-
Being-H0.7: A Latent World-Action Model from Egocentric Videos
Being-H0.7 adds future-aware latent reasoning to direct VLA policies via dual-branch alignment on latent queries, matching world-model benefits at VLA efficiency.
-
Gen4U: Unifying Video Generation and Understanding via Diffusion
Frozen video diffusion models, probed at optimal depth and noise levels, produce representations competitive with discriminative encoders across semantic and geometric video tasks in a single forward pass.
-
Neural Voxel Dynamics: Learning Implicit 3D Physics via Volumetric Feature Advection
A self-supervised framework learns implicit 3D physics by lifting V-JEPA features into voxels and performing volumetric feature advection conditioned on actions.
-
P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture
P-JEPA enables long-form procedural video understanding by predicting pooled masked latent vectors in a dense frame-aligned action space, achieving SOTA fine-grained action classification on EgoExo4D with an order of magnitude fewer parameters than LLM methods.
-
Active Inference as the Test-Time Scaling Law for Physical AI Agents
Active inference supplies a test-time scaling law for physical AI agents by framing policy updates as soft Bayesian inference that reduces expected prediction errors, with a variational solution shown to outperform Q-learning and Bayesian RL in driving simulations.
-
GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
GLINT introduces sparsely gated alignment and dense feature regularization on top of DINOv3 and V-JEPA encoders to enable query-specific zero-shot grounding and segmentation in 2D CXR and 3D CT.
-
Beyond Task Success: Behavioral and Representational Diagnostics for WAM and VLA
Empirical study introduces behavioral and representational diagnostics showing architecture-dependent gains in object targeting and predictive structure for WAMs over VLAs on LIBERO and RoboTwin2.0.
-
How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction
TrajPilot predicts candidate future trajectories from egocentric context and uses them to condition action prediction in an embedding space, outperforming VLM and planner baselines on Ego-Exo4D, Ego4D, and other datasets with gains increasing at longer horizons.
-
Latent Video Prediction Learns Better World Models
Latent prediction video models exhibit a distinct robustness profile across corruption, occlusion, fine-grained discrimination, and temporal sensitivity compared to other self-supervised video models when used as world models.
-
EgoExo-WM: Unlocking Exo Video for Ego World Models
Method converts exocentric videos to egocentric format via body-pose extraction and kinematics to improve egocentric world-model prediction and planning.
-
SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models
SARA introduces semantic saliency to guide relational alignment in video diffusion models, improving text following and motion quality over prior alignment methods.
-
3D MRI Image Pretraining via Controllable 2D Slice Navigation Task
Converting 3D MRI volumes into action-conditioned 2D slice navigation sequences offers a complementary self-supervised pretraining signal for learning anatomical and spatial representations.
-
Lifting Embodied World Models for Planning and Control
Planning with a frozen egocentric world model is lifted from 48-dim joint actions to a few 2D goal waypoints via a trained policy, reducing CEM error reduction by 3.8x.
-
Einstein World Models
Einstein World Models integrate visual rollouts from a callable world-module into LLM reasoning traces to support complex thought beyond language.
-
Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI
Kairos learns and maintains control-sufficient world states via a cross-embodiment curriculum, hybrid linear temporal attention, and deployment-aware co-design for Physical AI.
-
How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position
The paper proposes an L0-L7 evidential ladder for evaluating world models in embodied decision-making, prioritizing interventional action fidelity and policy optimization utility over visual plausibility.
-
Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis
Video foundation models encode intuitive physics knowledge that is strongest in V-JEPA at intermediate-to-late layers and depends on pretraining type and probe design.
-
HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning
HQ-JEPA combines JEPA-style predictive self-supervision with cross-modal alignment and a SWAP-test-based quantum fidelity loss for learning representations from paired remote sensing imagery, reporting competitive results on GeoBench tasks.
-
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
CoMET achieves strong multimodal classification performance by composing frozen modality encoders, PCA compression, and tabular foundation models without any training, reaching state-of-the-art on diverse benchmarks including large-scale hierarchical tasks.
-
Nano World Models: A Minimalist Implementation of Future Video Prediction
Nano World Models supplies a unified minimalist codebase and evaluation framework for studying diffusion forcing in video prediction across control, games, and robot domains.
-
Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives
Empirical tests show that factorized world-model with hard-region-weighted latent dynamics improves ImageNet-100 by 5.92 and SSv2 by 3.21 points over baseline in mixed-dataset pretraining while staying within 0.3 points on Diving-48.
-
Representation Without Reward: A JEPA Audit for LLM Fine-Tuning
An empirical audit of 22 JEPA-style training auxiliaries on Llama-3.2-1B fine-tuning for regex generation finds no statistically significant task improvement after multiple-testing correction, even when auxiliaries visibly alter hidden-state geometry.
-
Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
Semantic latent spaces from pretrained encoders outperform reconstruction-based spaces for robotic world models on planning and downstream policy performance.
-
InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
InternVideo3 introduces Multimodal Contextual Reasoning and M^2LA attention to enable closed-loop evidence accumulation in long-video understanding and agentic tool use, reporting strong benchmark results.
-
Motif-Video 2B: Technical Report
Motif-Video 2B reaches 83.76% on VBench, outperforming a 14B-parameter model with 7x fewer parameters and far less training data through shared cross-attention and a three-part backbone.
-
FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation
FROST-STA ranks second in the Ego4D Short-Term Object Interaction Anticipation challenge with 5.13 mAP by adapting frozen V-JEPA features with object-centric heads and ensembling.
-
TAP-JEPA: Frozen Future-Latent Probing and Two-Stage Score Fusion for EPIC-KITCHENS-100 Action Anticipation
TAP-JEPA applies frozen V-JEPA features, latent future prediction, and two-stage fusion of attentive probes to reach 27.91% MT5R and second place on the EK-100 action anticipation leaderboard.
-
JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026
JFAA freezes a JEPA future-prediction model, adds a lightweight probe and ensemble, and wins the 2026 EK-100 action anticipation challenge.
-
VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026
VISTA wins first place on the Ego4D Short-Term Object Interaction Anticipation challenge by combining spatial object proposals with temporal context via feature modulation and ROI fusion, followed by ensembling.