Pith. sign in

REVIEW 3 major objections 3 minor 47 references

The Structured Dynamics Model separates camera motion from object motion by predicting future features with two recurrent tokens, a separation learnable from frozen image features with only weak synthetic labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:01 UTC pith:PAZGZ5ET

load-bearing objection Honest, careful empirical paper whose central 'self-supervised structure' claim leans heavily on weak synthetic labels; the probe results are real but the decomposition is externally anchored. the 3 major comments →

arxiv 2607.21576 v1 pith:PAZGZ5ET submitted 2026-07-23 cs.CV

Self-Supervised Learning of Structured Dynamics from Videos

Lukas Knobel , Andrew Zisserman , Yuki M. Asano This is my paper
classification cs.CV
keywords self-supervised learningvideo representation learningmotion decompositionfuture feature predictionfrozen image featuresprimary and residual motion tokenscamera motionobject motion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a frozen pretrained image transformer can be turned into a structured video-dynamics representation. It introduces the Structured Dynamics Model (SDM), which predicts a future frame's features from the previous frame's features through two sequential compensation stages: a primary token explains the dominant source of change, and a residual token captures what remains. Training combines self-supervised future-feature prediction on real videos with weak synthetic labels indicating only whether the camera or the scene is static. The resulting tokens specialize by dominant motion regime, and linear probes on the new ProbeMotion suite decode camera motion, object motion, and action categories. If correct, this means dense 3D supervision and full video pretraining are not necessary to obtain useful, interpretable motion structure from image features.

Core claim

The paper's central claim is that temporal change in video can be organized into primary and residual motion components entirely in the feature space of a frozen image transformer. The primary stage extracts a recurrent token from an adjacent feature pair and compensates the source features; the residual stage does the same on the leftover mismatch. Selective weak supervision — static-scene samples bypass the residual stage, static-camera samples regularize the primary stage toward leaving the source unchanged — anchors the semantics of the two tokens. The authors show that the tokens specialize: the primary token carries camera motion in moving-camera videos and object motion in near-static

What carries the argument

The central mechanism is the two-stage compensate-and-refine feature prediction loop. Given consecutive feature maps, a primary motion extractor updates a recurrent token p; a predictor applies p to the source feature map to produce a primary-compensated estimate. A residual extractor compares that estimate with the target to update token r, and a residual predictor refines the estimate. The training objective selectively applies the feature-prediction loss: only the final prediction for unlabeled or dynamic videos, the primary output directly for static scenes, and a regularization that encourages the primary stage to leave the source unchanged for static cameras. That selective loss is wha

Load-bearing premise

The load-bearing premise is that weak scene-level labels from synthetic videos — static scene versus static camera — transfer well enough to real, unlabeled videos to anchor the intended primary/residual decomposition; if that transfer fails, the reported token specialization and probing gains would not generalize to in-the-wild videos.

What would settle it

Retrain SDM with the synthetic weak labels withheld and probe token roles on the synthetic motion benchmark: if the primary token no longer reliably beats the residual token for camera motion, and the residual no longer beats primary for object motion, then the decomposition is imposed by the synthetic anchor rather than learned from prediction. A second check: apply the trained tokens to real videos with accurate camera-pose ground truth and see whether the primary token's camera-motion decoding accuracy collapses relative to synthetic results.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A frozen image backbone plus small recurrent modules can produce structured motion tokens that linear probes decode into 3D camera motion, 2D and 3D object motion, and action classes.
  • Structured two-token prediction outperforms naive global descriptors such as CLS tokens, average-pooled features, and single delta-token models on most of the seven ProbeMotion tasks.
  • The primary/residual split adapts to the dominant motion source: primary carries camera motion in moving-camera videos and object motion in near-static-camera videos, while residual carries leftover object dynamics.
  • The approach matches or beats much larger geometry-supervised representations on several probes while using far weaker supervision, suggesting that explicit dense 3D labels are not required to recover useful motion structure.
  • Longer temporal context improves the tokens monotonically, even beyond the training horizon, indicating the recurrent tokens accumulate motion information rather than just comparing frame pairs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the two-token decomposition generalizes broadly, it could replace single entangled latent actions in world models, giving an interpretable camera-versus-object separation that downstream planners could exploit.
  • Editorial inference: the synthetic weak labels are the component most likely to limit scaling; a fully self-supervised objective that enforces separation through cross-transition consistency would be a natural and decisive next experiment.
  • Editorial inference: because motion tokens transfer between clips in the qualitative experiments, they behave like reusable transformation parameters, suggesting applications in video editing and motion transfer, though only short-horizon effects are demonstrated.
  • Editorial inference: the drift observed in long-horizon extrapolation may reflect the short training horizon rather than a token-capacity limit; training with longer clips could extend the reliable prediction horizon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper asks whether frozen image ViT features can be remodeled into structured motion representations without dense 3D supervision. It introduces the Structured Dynamics Model (SDM): a recurrent two-stage model on top of frozen DINOv2 features that predicts future feature maps. The primary stage extracts a recurrent token p_t from consecutive feature pairs and compensates the source feature map; the residual stage extracts r_t from the remaining mismatch and refines the prediction. Training uses future-feature prediction on a mix of synthetic Kubric data (with weak labels indicating static scene or static camera) and unlabeled real videos (SSv2, DL3DV). The static-scene label bypasses the residual stage; the static-camera label regularizes the primary stage to leave features unchanged. Evaluation introduces ProbeMotion, a suite of linear-probing tasks on Kubric, DL3DV, CameraBench, static DAVIS2017/YouTubeVOS, and SSv2-110k. SDM is compared to frozen DINOv2 descriptors, DeltaTok, and strongly supervised VGGT/DA3/Pi3X. The paper reports that SDM outperforms frozen-feature baselines and DeltaTok on most probes, and is competitive with or better than supervised models on several tasks. It also analyzes token specialization, temporal context, and ablates supervision components.

Significance. If the claims hold, the result is valuable: a simple recurrent feature-prediction module on frozen DINOv2 features can produce linearly decodable camera/object motion representations competitive with much larger, densely supervised geometry models. The paper's strengths include a carefully controlled evaluation protocol (three seeds, probe learning-rate sweeps, multi-layer feature baselines, ablations, token-swapping tests, and cross-backbone generalization) and an honest limitations section. However, the central primary/residual decomposition is actively anchored by weak synthetic labels, and the ablations show that those labels are necessary for the reported specialization. The significance is therefore conditional on accepting a weakly supervised hybrid, and the unqualified "Self-Supervised Learning" framing overstates what is demonstrated.

major comments (3)
  1. [§2.3, §B.5, Table 13a] The paper's title and framing imply that the primary/residual decomposition emerges from self-supervised future-feature prediction. The ablations in Table 13a show the opposite: removing static-scene supervision makes the residual token nearly as predictive of camera motion as the primary token (0.21 vs 0.18), and removing static-camera supervision makes the primary token more predictive of object motion than the residual token (0.14 vs 0.38). Thus each weak label is individually load-bearing for the specialization that the paper interprets as learned structure. The manuscript never trains a fully unlabeled variant, so it is unknown whether any structure appears without the synthetic anchors. Appendix C acknowledges that the decomposition is 'anchored by weak scene-level supervision on synthetic data,' but this is not reflected in the title. Please either add a no-label control (or a pri
  2. [§3.2, Table 2; §3.4, Table 5] The headline comparisons in Table 2 use a per-benchmark token choice that is itself derived from the primary/residual hypothesis under test. For example, Kubric object motion in dynamic-camera scenes is probed with r, while all other tasks use p; the choice is motivated by the same decomposition the paper aims to establish. The token-swapping results in Table 5 are a useful safeguard, but they do not fully fix the comparison because the main table never shows baseline methods under an analogous 'best-token' protocol. Please either include both p and r (or the concatenation [p,r]) in the main ProbeMotion table, or adopt a fixed selection rule (e.g., always p) and present the adaptive choice as an additional analysis. This is needed to decouple the reported gains from the selection protocol.
  3. [§3.4, §B.1, §A.2] The generalization of the primary/residual structure to real videos is only indirectly demonstrated. For static DAVIS2017/YouTubeVOS and SSv2, the dominant motion source is inferred from VGGT camera-motion statistics, and token specialization is then read off from probe accuracy. The paper acknowledges this in Section C, but a direct test on a small labeled real-video set (e.g., using available camera poses and object masks, or human annotations of dominant motion) would substantially strengthen the central transfer claim. Without it, the claim that the tokens 'adapt to the dominant sources of change in each video' remains an interpretation of probe rankings rather than a measured property.
minor comments (3)
  1. [Abstract and tables] The abstract contains 'usingsubstantiallyweaker' (missing spaces); Table 1's header 'DA VIS2017 YouTubeVOS Bench110k' is broken; and 'A VG' is used inconsistently in Tables 2, 8, and 9. Please fix these formatting issues.
  2. [§3.2] The static-camera regularization loss could be given a numbered equation; currently it is described only in prose. Also, please make sure λ_reg is consistently reported as 0.5 in the main text, matching Table 13e.
  3. [Figure 1] The left panel contains stray numbers ('15 2 14 23 14') that are not explained in the caption; please clean this up or explain the notation.

Circularity Check

1 steps flagged

Weak synthetic labels, not self-supervision, force the primary/residual split; the 'structured dynamics' claim is partly the training objective restated, though real-video transfer is independently tested.

specific steps
  1. fitted input called prediction [§2.3 Training objective; §3.4 Structured motion; §B.5/Table 13a]
    "To align the architectural decomposition with the intended semantics of primary and residual motion, we use two weak scene-level annotations, static scene and static camera, and selectively apply the loss. Static scenes: ... the primary component should explain the full transition, so we bypass the residual stage and supervise the output of the primary compensation directly: L=LMSE(f>_t, ft). Static cameras: ... the primary stage is regularized to leave the source representation unchanged while the residual stage captures scene dynamics."

    The Kubric 'static scene' samples are exactly moving-camera-only clips and 'static camera' samples are exactly object-motion-only clips (§A.1). The static-scene loss therefore directly fits p to the camera-induced transition, and the static-camera regularizer forces p to encode no change while routing object motion to r. §3.4 then reports the resulting asymmetry as evidence that 'the learned roles support the primary/residual interpretation.' Table 13a shows the labels are individually necessary (without static-camera supervision, p beats r for object motion, 0.14 vs 0.38). So the p/r specialization is partly a restatement of the training objective, not an emergent SSL discovery; §C concedes the decomposition is 'anchored by weak scene-level supervision on synthetic data.' Real-video probe

full rationale

The paper is not globally circular: the main engineering contribution—two recurrent motion tokens on a frozen DINOv2 backbone, trained by future-feature prediction and evaluated on external ProbeMotion benchmarks against supervised baselines—is self-contained and independently checked. The DL3DV, CameraBench, static DAVIS/YouTubeVOS, and SSv2 results do not reduce to the training labels, and the ablations are reported honestly. However, the central 'structured dynamics' claim has a circular component. In Kubric, the weak labels are not arbitrary side information: 'static scene' means only camera motion is present, and the loss directly supervises the primary stage to fit that transition; 'static camera' means only object motion is present, and the regularizer forces the primary stage to be identity while the residual stage carries the object motion. The observed p/r specialization is therefore substantially built into the training objective, and Table 13a confirms that removing either label destroys the specialization. Section 3.4 presents this as a learned finding, and the evaluation protocol selects p or r according to the same expected specialization. The paper's own §C acknowledges the limitation. No load-bearing self-citation was found; the Asano-authored references [27,28,33] appear only in related-work descriptions and are not used to justify the central derivation. Overall, the independent real-video generalization prevents a higher score, but the decomposition claim itself earns a 6 on the circularity scale.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on domain assumptions about feature-space sufficiency, single-token representability, synthetic-to-real transfer of weak labels, VGGT-filtered targets, and linear-probe measurement; the two learned tokens are internal latents without external falsifiable evidence.

free parameters (4)
  • λreg (static-camera regularization weight) = 0.5
    Set by scanning 0.1, 0.25, 0.5, 1.0 and choosing a compromise between Kubric object-motion MSE and SSv2-110k accuracy (Table 13e). This value affects the primary/residual specialization, a central claim.
  • Loss function choice (MSE vs L1) = MSE
    MSE improves Kubric object-motion probing (0.30→0.19) at slight cost on real-video probes; this choice is motivated by one benchmark (Table 13d).
  • Number of registers nreg in motion extractor = 4
    Architectural hyperparameter set without ablation; part of the motion extractor capacity.
  • Training data mixture ratio (Kubric/SSv2/DL3DV) = 0.5/0.25/0.25
    Fixed mixture of synthetic and real videos; not ablated, affects transfer of the decomposition to real data.
axioms (5)
  • domain assumption Frozen DINOv2 features contain sufficient information to predict future features and decode motion.
    The entire method operates in the frozen feature space (Section 2.1); if features discard motion cues, the model cannot recover them.
  • domain assumption A single primary token p_t can represent the dominant source of temporal change in any clip.
    The architecture forces all dominant change through one D-dimensional token (Section 2.2); scenes with multiple comparable motions would violate this.
  • domain assumption Weak scene-level labels on synthetic Kubric (static scene vs static camera) transfer to real videos.
    Training uses these labels (Section 2.3); evaluation on real videos assumes the induced decomposition generalizes. Section C acknowledges this dependency.
  • domain assumption VGGT-estimated camera motion is reliable enough to construct static-camera subsets of DAVIS/YouTubeVOS.
    Static-camera benchmarks are filtered with VGGT (Section A.2); errors in these estimates add label noise, acknowledged in Section C.
  • domain assumption Linear probes measure the quality of the learned motion representations.
    All conclusions are based on linear probes (Section 3.2); nonlinear decodability or other probe families are not assessed. Section C notes this limitation.
invented entities (2)
  • Primary motion token p_t no independent evidence
    purpose: Encodes the dominant source of temporal change (camera motion in dynamic-camera videos, object motion in static-camera videos).
    Introduced as a new latent variable; evidence for its semantics is the paper's own probing (Table 5) and qualitative extrapolation; no external falsifiable handle is provided.
  • Residual motion token r_t no independent evidence
    purpose: Encodes dynamics left unexplained by primary compensation, e.g., object motion under camera motion.
    Same as above; internal latent whose only evidence comes from in-paper probes and visualizations.

pith-pipeline@v1.3.0-alltime-deepseek · 22896 in / 15378 out tokens · 139908 ms · 2026-08-01T07:01:21.651526+00:00 · methodology

0 comments
read the original abstract

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent or with unstructured, spatially dense transition tokens. Training combines self-supervised learning on real video with weak supervision of scene dynamics on synthetic Kubric data. We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features, and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision. These results suggest that pretrained image models can be readily repurposed into structured video-dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

Figures

Figures reproduced from arXiv: 2607.21576 by Andrew Zisserman, Lukas Knobel, Yuki M. Asano.

Figure 1
Figure 1. Figure 1: The Structured Dynamics Model (SDM) remodels frozen image features into struc￾tured motion tokens. Left: A frozen image encoder extracts frame features, and a recurrent SDM is trained by self-supervised future-feature prediction. SDM structures temporal change into a primary motion token p for the dominant source of change and a residual motion token r for remaining dynamics. Right: Linear probes on ProbeM… view at source ↗
Figure 2
Figure 2. Figure 2: 2 [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the structured dynamics model shown for T=3. For each transition, SDM predicts the target frozen feature map ft from the previous one ft−1 through two sequential compensation stages. The primary stage extracts a recurrent motion token pt from the feature pair (ft−1, ft) and uses it to produce a primary-compensated feature map f > t , explaining the dominant change. The residual stage extracts r… view at source ↗
Figure 3
Figure 3. Figure 3: Stage-wise structured feature prediction. We visualize feature-map error to the target after primary and residual compensation on a sample from DAVIS2017. Errors are highlighted in red. The primary stage corrects the dominant global change, including the camera-induced movement of the edges of the track. Notably, primary motion compensation increases the error around the independently moving car. The resid… view at source ↗
Figure 4
Figure 4. Figure 4: Motion extrapolation. We repeatedly apply the last motion token from the observed frames to predict future feature maps on a sample from the DAVIS2017 test split. Visualization is done by PCA projection to RGB after normalization. Black boxes mark the last observed ground-truth object position, which is repeated for the extrapolated feature maps. The extrapolated object features follow the object’s motion … view at source ↗
Figure 5
Figure 5. Figure 5: Latent motion swapping. We apply motion tokens from a source clip to the feature map of a target clip with a different object, both from the Kubric test set, and visualize the resulting features by PCA projection to RGB after normalization. Black boxes mark the initial object position. The target feature map follows the transferred motion over short horizons and drifts at longer horizons. predictive object… view at source ↗
Figure 6
Figure 6. Figure 6: Kubric examples. Example sequences from our generated Kubric training data. The three subsets provide weak scene-level supervision for disentangling primary and residual dynamics: moving-camera/static-scene sequences contain only camera-induced motion, static-camera/dynamic￾scene sequences contain only object motion, and moving-camera/dynamic-scene sequences combine both sources of motion. Examples are sho… view at source ↗
Figure 7
Figure 7. Figure 7: shows an additional visualization of the error between predicted and ground-truth feature maps. 𝑓t > 𝑓t 𝑓t ≫ 𝑓t Error after 𝐫𝐞𝐬𝐢𝐝𝐮𝐚𝐥 𝐦𝐨𝐭𝐢𝐨𝐧 target frame t Error after 𝐩𝐫𝐢𝐦𝐚𝐫𝐲 𝐦𝐨𝐭𝐢𝐨𝐧 source frame t − 1 𝑓t 𝑓𝑡−1 Error without any correction [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 15 linked inside Pith

  1. [1]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985,

    Pith/arXiv arXiv

  2. [2]

    Back to the features: Dino as a foundation for video world models.arXiv preprint arXiv:2507.19468, 2025

    Federico Baldassarre, Marc Szaffraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann LeCun, Patrick Labatut, Maximilian Seitzer, and Piotr Bojanowski. Back to the features: Dino as a foundation for video world models.arXiv preprint arXiv:2507.19468, 2025. 10

    Pith/arXiv arXiv 2025

  3. [3]

    Revisiting feature prediction for learning visual representations from video.Transactions on Machine Learning Research, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video.Transactions on Machine Learning Research, 2024. 10

    2024

  4. [4]

    VFMF: World modeling by forecasting vision foundation model features.arXiv preprint arXiv:2512.11225,

    Gabrijel Boduljak, Yushi Lan, Christian Rupprecht, and Andrea Vedaldi. VFMF: World modeling by forecasting vision foundation model features.arXiv preprint arXiv:2512.11225,

    arXiv

  5. [5]

    TIPSv2: Advancing vision- language pretraining with enhanced patch-text alignment.arXiv preprint arXiv:2604.12012,

    Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen, Arjun Karpur, Ye Xia, Sahil Dua, Tanmaya Dabral, Guangxing Han, Bohyung Han, et al. TIPSv2: Advancing vision- language pretraining with enhanced patch-text alignment.arXiv preprint arXiv:2604.12012,

    Pith/arXiv arXiv

  6. [6]

    A short note on the kinetics- 700 human action dataset.arXiv preprint arXiv:1907.06987, 2019

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics- 700 human action dataset.arXiv preprint arXiv:1907.06987, 2019. 18

    Pith/arXiv arXiv 1907

  7. [7]

    Scaling 4d representations.arXiv preprint arXiv:2412.15212, 2024

    João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahen- dran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, et al. Scaling 4d representations.arXiv preprint arXiv:2412.15212, 2024. 9

    Pith/arXiv arXiv 2024

  8. [8]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. InThe Twelfth International Conference on Learning Representations, 2024. 4, 8, 18, 24

    2024

  9. [9]

    Probing the 3d awareness of visual foundation models

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21795–21806, 2024. 9

    2024

  10. [10]

    Scalable pre-training of large autoregressive image models

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Ángel Bautista, Vaishaal Shankar, Alexander T Toshev, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. InInternational Conference on Machine Learning, pages 12371– 12384, 2024. 9

    2024

  11. [11]

    Multimodal autoregressive pre-training of large vision encoders

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor G Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregressive pre-training of large vision encoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9641–9654, 2025. 1

    2025

  12. [12]

    Learning latent action world models in the wild.arXiv preprint arXiv:2601.05230,

    Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, and Michael Rabbat. Learning latent action world models in the wild.arXiv preprint arXiv:2601.05230,

    arXiv

  13. [13]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. InProceedings of the IEEE international conference on computer vision, pages 5842– 58...

    2017

  14. [14]

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Meh...

    2022

  15. [15]

    Siamese masked autoencoders.Advances in Neural Information Processing Systems, 36:40676–40693, 2023

    Agrim Gupta, Jiajun Wu, Jia Deng, and Fei-Fei Li. Siamese masked autoencoders.Advances in Neural Information Processing Systems, 36:40676–40693, 2023. 3, 10

    2023

  16. [16]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. 8, 9, 24

    2022

  17. [17]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. InEuropean Conference on Computer Vision, pages 289–305. Springer,

  18. [18]

    VGGT4D: Mining motion cues in visual geometry transformers for 4d scene reconstruction.arXiv preprint arXiv:2511.19971, 2025

    Yu Hu, Chong Cheng, Sicheng Yu, Xiaoyang Guo, and Hao Wang. VGGT4D: Mining motion cues in visual geometry transformers for 4d scene reconstruction.arXiv preprint arXiv:2511.19971, 2025. 10

    arXiv 2025

  19. [19]

    A frame is worth one token: Efficient generative world modeling with delta tokens.arXiv preprint arXiv:2604.04913, 2026

    Tommie Kerssies, Gabriele Berton, Ju He, Qihang Yu, Wufei Ma, Daan de Geus, Gijs Dubbel- man, and Liang-Chieh Chen. A frame is worth one token: Efficient generative world modeling with delta tokens.arXiv preprint arXiv:2604.04913, 2026. 2, 3, 6, 7, 10, 18, 24

    Pith/arXiv arXiv 2026

  20. [20]

    Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025. 1, 6, 9, 24

    Pith/arXiv arXiv 2025

  21. [21]

    Towards understanding camera motions in any video.arXiv preprint arXiv:2504.15376, 2025

    Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, et al. Towards understanding camera motions in any video.arXiv preprint arXiv:2504.15376, 2025. 5, 24

    Pith/arXiv arXiv 2025

  22. [22]

    DL3DV-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. DL3DV-10k: A large-scale scene dataset for deep learning-based 3d vision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160–22169, 2024. 5, 24

    2024

  23. [23]

    4RC: 4d recon- struction via conditional querying anytime and anywhere.arXiv preprint arXiv:2602.10094,

    Yihang Luo, Shangchen Zhou, Yushi Lan, Xingang Pan, and Chen Change Loy. 4RC: 4d recon- struction via conditional querying anytime and anywhere.arXiv preprint arXiv:2602.10094,

    Pith/arXiv arXiv

  24. [24]

    V-JEPA 2.1: Unlocking dense features in video self-supervised learning.arXiv preprint arXiv:2603.14482, 2026

    Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-JEPA 2.1: Unlocking dense features in video self-supervised learning.arXiv preprint arXiv:2603.14482, 2026. 4, 10

    Pith/arXiv arXiv 2026

  25. [25]

    DINOv2: Learning robust visual features without supervision.Transactions on Machine Learning Research, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.Transactions on Machine Learning Research, 2023. 1, 9, 17

    2023

  26. [26]

    The 2017 davis challenge on video object segmentation.arXiv:1704.00675,

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv:1704.00675,

    Pith/arXiv arXiv 2017

  27. [27]

    Time does tell: Self-supervised time-tuning of dense image representations

    Mohammadreza Salehi, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Time does tell: Self-supervised time-tuning of dense image representations. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 16536–16547, 2023. 10

    2023

  28. [28]

    MoSiC: Optimal-transport motion trajectory for dense self- supervised learning

    Mohammadreza Salehi, Shashanka Venkataramanan, Ioana Simion, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. MoSiC: Optimal-transport motion trajectory for dense self- supervised learning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 6541–6551, 2025. 10

    2025

  29. [29]

    Oriane Siméoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. DINOv3. arXiv preprint arXiv:2508.10104, 2025. 1, 8, 18, 24

    Pith/arXiv arXiv 2025

  30. [30]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. 4

    2024

  31. [31]

    VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.Advances in neural information processing systems, 35:10078–10093, 2022. 9

    2022

  32. [32]

    SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, local- ization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Al- abdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, local- ization, and dense features.arXiv preprint arXiv:2502.14786, 2025. 1

    Pith/arXiv arXiv 2025

  33. [33]

    Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video

    Shashanka Venkataramanan, Mamshad Nayeem Rizve, Joao Carreira, Yuki M Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. InThe Twelfth International Conference on Learning Representations, 2024. 10

    2024

  34. [34]

    PooDLe: Pooled and dense self-supervised learning from naturalistic videos

    Alex N Wang, Christopher Hoang, Yuwen Xiong, Yann LeCun, and Mengye Ren. PooDLe: Pooled and dense self-supervised learning from naturalistic videos. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025. 10

    2025

  35. [35]

    VGGT: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025. 1, 9, 10, 19, 24

    2025

  36. [36]

    VideoMAE v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE v2: Scaling video masked autoencoders with dual masking. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14549–14560,

  37. [37]

    Continuous 3d perception model with persistent state

    Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10510–10522, 2025. 10

    2025

  38. [38]

    DUSt3R: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3d vision made easy. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20697–20709, 2024. 1, 10

    2024

  39. [39]

    pi3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025

    Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. pi3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025. 1, 6, 9, 24

    Pith/arXiv arXiv 2025

  40. [40]

    CroCo: Self-supervised pre-training for 3d vision tasks by cross-view completion.Advances in Neural Information Processing Systems, 35:3502–3516, 2022

    Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. CroCo: Self-supervised pre-training for 3d vision tasks by cross-view completion.Advances in Neural Information Processing Systems, 35:3502–3516, 2022. 1, 3

    2022

  41. [41]

    YouTube-VOS: Sequence-to-sequence video object segmentation

    Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. YouTube-VOS: Sequence-to-sequence video object segmentation. InProceedings of the European conference on computer vision (ECCV), pages 585–601, 2018. 5, 24 13

    2018

  42. [42]

    YouTube-VOS: A large-scale video object segmentation benchmark.arXiv preprint arXiv:1809.03327, 2018

    Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. YouTube-VOS: A large-scale video object segmentation benchmark.arXiv preprint arXiv:1809.03327, 2018. 5, 24

    Pith/arXiv arXiv 2018

  43. [43]

    In pursuit of pixel supervision for visual pre-training.arXiv preprint arXiv:2512.15715, 2025

    Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Heng- shuang Zhao, and Hu Xu. In pursuit of pixel supervision for visual pre-training.arXiv preprint arXiv:2512.15715, 2025. 1

    arXiv 2025

  44. [44]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 8, 24

    2023

  45. [45]

    MonST3R: A simple approach for estimating geometry in the presence of motion

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. MonST3R: A simple approach for estimating geometry in the presence of motion. InThe Thirteenth International Conference on Learning Representa- tions, 2025. 10

    2025

  46. [46]

    DINO-WM: World models on pre-trained visual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann Lecun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. InInternational Conference on Machine Learning, pages 79115–79135. PMLR, 2025. 10

    2025

  47. [47]

    Recurrent video masked autoencoders

    Daniel Zoran, Nikhil Parthasarathy, Yi Yang, Drew A Hudson, João Carreira, and Andrew Zisserman. Recurrent video masked autoencoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17744–17755, 2026. 9 14 A Dataset and Evaluation Details Table 6:Evaluation tasks.Summary of the probing tasks used for each evaluat...

    2026