CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:QGN5DBXTrecord.jsonopen to challenge →
read the original abstract
Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points.
This paper has not been read by Pith yet.
Forward citations
Cited by 32 Pith papers
-
UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image
UnfoldArt uses a two-round structured debate between high-level semantic agents and low-level parameter agents, grounded in generated video, to infer articulation and reconstruct full articulated 3D objects including ...
-
UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image
UnfoldArt uses multi-agent debate grounded in vision-language and video models to infer articulation parameters and reconstruct full 3D objects including occluded parts from text or image inputs.
-
Point Tracking Improves World Action Models
JOPAT jointly models pixels, point tracks, and actions in a diffusion transformer and reports gains over pixel-only baselines on long-horizon robot tasks with occlusion and off-screen motion.
-
DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation
DeVI enables zero-shot physically plausible dexterous control by imitating synthetic videos via a hybrid 3D-human plus 2D-object tracking reward.
-
ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
ReImagine decouples human appearance from temporal consistency via pretrained image backbones, SMPL-X motion guidance, and training-free video diffusion refinement to generate high-quality controllable videos.
-
Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers
A model-free system uses 2D point trackers to achieve causal 6D pose tracking and incremental 3D reconstruction for multiple unseen rigid objects from RGB-D video, with recovery from complete occlusions.
-
SAM 2++: Tracking Anything at Any Granularity
SAM 2++ unifies video tracking across mask, box, and point granularities via task-specific prompts, a unified decoder, task-adaptive memory, and a new multi-granularity dataset, reporting state-of-the-art results.
-
Follow Your Track: Precise Skeleton Animation Controlled by 3D Trajectories
ACT is a trajectory-conditioned framework for topology-general skeletal animation that injects 3D point trajectories from monocular video into skeletons via a Routed Trajectory Injector for improved fidelity and tempo...
-
MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
Introduces a new task of goal-conditioned 3D point motion forecasting along with a 1.16M-video dataset, a 111-category benchmark, and a model that outperforms baselines while transferring to robotics and video generation.
-
4D Reconstruction from Sparse Dynamic Cameras
Presents a 3D track initialization method, depth-ordering regularization, and batch sampling for 4D reconstruction from sparse dynamic cameras, plus the LetCamsGo dataset showing gains in dynamic regions.
-
Training-Free Object-Agnostic Jam Detection in Fulfillment Centers
A training-free jam detection method monitors persistent occlusion of uniformly sampled reference points and reports 100% precision and 93.33% F1 on 1,069 videos.
-
DexSIM: Real-time Dexterous Simulation with Unified Causal Video Diffusion
DexSIM is a bi-directional video diffusion model with hand trajectory embedding and spatial memory cache for real-time dexterous hand-object simulation at 15 FPS.
-
The TIME Machine: On The Power of Motion for Efficient Perception
TIME is a motion-based embedding from point tracks, trained only on synthetic data via masked autoencoding, that matches state-of-the-art video model performance with up to 10,000x less training data.
-
Quantitative Video World Model Evaluation for Geometric-Consistency
PDI-Bench computes 3D projective residuals from segmented and tracked points to quantify geometric inconsistency in AI-generated videos.
-
ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation
ConsisVLA-4D adds cross-view semantic alignment, cross-object geometric fusion, and cross-scene dynamic reasoning to VLA models, delivering 21.6% and 41.5% gains plus 2.3x and 2.4x speedups on LIBERO and real-world tasks.
-
GenMatter: Perceiving Physical Objects with Generative Matter Models
A hierarchical probabilistic model with parallelized Gibbs sampling segments moving matter across random-dot, camouflaged-texture, and naturalistic-video domains, matching supervised baselines and human perceptual judgments.
-
Wiggle and Go! System Identification for Zero-Shot Dynamic Rope Manipulation
Wiggle and Go! uses system identification from rope motion observations to predict parameters that enable zero-shot goal-conditioned dynamic manipulation, achieving 3.55 cm accuracy on 3D target striking versus 15.34 ...
-
TrackerSplat: Exploiting Point Tracking for Fast and Robust Dynamic 3D Gaussians Reconstruction
TrackerSplat pre-positions 3D Gaussians using point tracking trajectories to handle large inter-frame displacements in dynamic scene reconstruction.
-
Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks
DenseMarks learns a canonical 3D embedding space for human head images by training a Vision Transformer with contrastive loss on pairwise point tracks from in-the-wild videos, plus landmark and segmentation supervision.
-
DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge
DreamVLA uses dynamic-region-guided world knowledge prediction, block-wise attention to disentangle information types, and a diffusion transformer for actions, reaching 76.7% success on real robot tasks and 4.44 avera...
-
Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations
RIGVid shows that filtered AI-generated videos can serve as effective supervision for complex robotic manipulation tasks without any real demonstrations.
-
PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation
PhysisForcing applies trajectory and relational alignment losses to DiT features in video models, improving physical plausibility on R-Bench, PAI-Bench, and EZS-Bench while raising closed-loop robotic success rates fr...
-
AnimaSpark: A Feed-Forward Method for Animating Arbitrary 3D Objects
AnimaSpark is a feed-forward pipeline that renders rigged 3D models to multi-layer images, generates video motion, tracks projected keypoints, and lifts 2D planar transforms into 3D skeletal animation.
-
IMPose: Interactive Multi-person Pose Estimation with Dynamic Correction Propagation
IMPose introduces dual-level (keypoint and instance) correction propagation with a trajectory bank to turn sparse annotations into dense multi-person pose trajectories in videos.
-
PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions
PhysHanDI achieves full 3D hand and non-rigid object reconstruction by simulating object deformations from hand-induced forces and refining hand models via inverse physics, outperforming prior methods in reconstructio...
-
GenMatter: Perceiving Physical Objects with Generative Matter Models
GenMatter is a generative hierarchical model that groups low-level motion and high-level features into particles and clusters representing independently moveable physical entities, validated across dot kinematograms, ...
-
MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models
MASS adds spatiotemporal motion signals and 3D grounding to VLMs and releases MASS-Bench, yielding physics-reasoning performance within 2% of Gemini-2.5-Flash after reinforcement fine-tuning.
-
Image-Guided Shape-from-Template Using Mesh Inextensibility Constraints
An unsupervised SfT approach using image observations and mesh inextensibility constraints reconstructs deforming 3D shapes 400x faster than prior unsupervised methods while handling severe occlusions better.
-
DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment
DriVerse is a generative model that simulates driving scenes from an image and trajectory using multimodal prompting and motion alignment, achieving better performance on nuScenes and Waymo datasets with minimal training.
-
WorldString: Actionable World Representation
Proposes WorldString, a differentiable neural model for the state manifold of actionable physical objects learned directly from 3D or video data as a building block for world models.
-
WorldString: Actionable World Representation
WorldString is a fully differentiable neural model for representing actionable object states learned from 3D sensor data.
-
Vision-Based Water Level and Flow Estimation
An integrated framework is proposed that combines SOTA vision models with statistical modeling and physical priors to improve accuracy in water level detection and flow estimation.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.