Pith. sign in

REVIEW 3 major objections 6 minor 46 references

DreamTraj claims the first direct decode of 6-DoF object trajectories from the internal attention maps of a frozen video diffusion model, needing only one RGB frame and an instruction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A single RGB image plus a language instruction predicts a metric 6-DoF object trajectory, decoded from the intermediate latents of a frozen video diffusion model at an early denoising step.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Worth serious review: the real-capture numbers support the core claim, but the readout-cell/step selection must be shown to be fold-disjoint before the SOTA tables can be trusted. the 3 major comments →

arxiv 2608.00486 v1 pith:XT5C2GF6 submitted 2026-08-01 cs.CV

DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents

Tongsheng Ding , Zhen Luo , Yixuan Yang , Boyu Wang , Luyang Xie , Jinyu Yang , Feng Zheng This is my paper
classification cs.CV
keywords 6-DoF trajectory predictionvideo diffusion latent readoutattention-based point trackingegocentric manipulationlanguage-conditioned motionflow matchingfrozen video diffusion model
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a video diffusion model's plan for a future scene is already written in its intermediate attention maps, before any video is rendered, and that a small trained readout can decode the manipulated object's metric 6-DoF trajectory from those maps. DreamTraj takes one RGB frame and a natural-language instruction, runs a frozen image-to-video model to an early denoising step, and reads attention tracks and pooled features with a 7.1M-parameter flow-matching reader. To train it, the authors contribute MOVE, 5,038 human-inspected egocentric trajectories paired with fine-grained instructions. They report state-of-the-art translation and rotation against forecasters that receive extra video, depth, or CAD inputs, and a 4.6x speedup over generating a video and extracting motion from pixels. The pay-off, if the central claim holds, is a minimal-input motion predictor that gets its dynamics from internet-scale video pretraining instead of from scarce interaction data.

Core claim

The central claim is that a clip-specific object-motion signal emerges in the intermediate features of a frozen image-to-video diffusion transformer, and can be decoded into a metric SE(3) trajectory without ever synthesizing a frame. The paper reads query-key attention from one cell of Wan2.2 (block 28, head 32) at denoising step 16 of 40, treats the soft matches as a point track of the object, and pools hidden states from three blocks for scene context. A 7.1M-parameter flow-matching DiT reader turns these into 13 scale-free 9-D pose tokens, and a monocular depth estimate anchors them to metric units. The reader is trained on both recorded and generated-video features with a domain flag, t

What carries the argument

The load-bearing object is the query-key attention readout cell (block 28, head 32) of the frozen Wan2.2 image-to-video backbone, whose head-averaged cosine similarity against frame 0 forms a soft per-frame track of the manipulated object. Read at denoising step 16 of 40, just after the model's high-to-low-noise expert switch, this cell yields centroid, spread, and in-plane rotation cues; mean-pooled hidden states from blocks 0/19/39 add scene geometry. The trainable component is a flow-matching DiT that jointly denoises all 13 relative pose tokens, so total displacement is decided holistically rather than accumulated step by step. Scale is restored only at the output from one depth estimate

Load-bearing premise

The whole readout rests on one fixed attention head inside the frozen video model continuing to track the manipulated object on scenes it has never seen, a property the paper checks with a correlation margin on 955 clips rather than on held-out folds.

What would settle it

Take the frozen Wan2.2 backbone, run it on 100 held-out egocentric clips from a corpus not among the six used to build MOVE, and compute the permutation-controlled margin of Eq. 3 for block 28, head 32 against the recorded 2-D tracks. If the average margin is not strongly positive, the fixed attention cell is not tracking the object on novel scenes and the trajectory signal is gone; one could also ablate the depth anchor to isolate scale recovery.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Object motion can be predicted at a fraction of the cost of video generation because the expensive render-and-measure loop is bypassed.
  • Predictors no longer need privileged depth, CAD, or multi-frame context at inference; the frozen video prior supplies the dynamics.
  • The trajectory is available as a compact SE(3) sequence in the camera frame, directly consumable by off-the-shelf manipulation controllers.
  • The latent readout makes the video model itself a motion prior rather than a video synthesizer, so the same backbone can serve both tasks.
  • A fine-grained language-to-motion dataset is now available for training and benchmarking.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The readout is probably not unique to Wan2.2: the paper's own evidence that one cell encodes an object track suggests other video-diffusion transformers harbor analogous cells, so the recipe (find the best block-head pair by a permutation-controlled margin, read at the expert switch) may transfer to newer backbones without retraining the reader from scratch.
  • The expert-switch discontinuity is a general design hint: the paper's sweep shows the motion signal appears discontinuously when the denoiser changes noise-expert regimes, which could guide where to probe other latent properties.
  • A testable extension is to apply the same latent readout to articulated or deformable objects by reading multiple query cells, since the current pipeline is limited to a single rigid object grounded in the first frame.
  • If the frozen prior already commits to a plausible motion, the reader could be trained in a more self-supervised loop on generated-video pseudo-labels, reducing the need for human-inspected 6-DoF supervision further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DreamTraj, a method that predicts a 6-DoF object trajectory from a single RGB frame and a natural-language instruction. Instead of generating a full video and extracting motion from pixels, DreamTraj runs a frozen image-to-video diffusion backbone (Wan2.2) to an early denoising step and reads two internal signals: query-key attention tracks from a single attention cell and pooled hidden states. A lightweight 7.1M-parameter flow-matching Reader decodes these features into relative 6-DoF pose tokens, which are lifted to metric scale using a monocular depth estimate. The paper also presents the MOVE dataset, 5,038 human-screened egocentric manipulation trajectories with per-clip language instructions, and evaluates DreamTraj with five-fold cross-validation against two privileged-input baselines, showing accuracy improvements and a 4.6x speedup over a generate-then-extract pipeline. The central claim is that this is the first approach to directly decode 6-DoF object trajectories from intermediate video-diffusion latents.

Significance. If the claims hold, the paper makes a meaningful contribution: it demonstrates that video-diffusion internal features can serve as an efficient, implicit motion prior for 3D trajectory prediction, removing the need for privileged depth, video, or CAD inputs at inference while avoiding full video synthesis. The MOVE dataset, with fine-grained language annotations and grouped cross-validation over source clips, is a useful resource. The paper provides quantitative comparisons on a real-capture, real-ground-truth subset (Table 1a), which supports the accuracy claim independently of the generated-video loop, and informative ablations (Table 4) that validate the contributions of the two readout signals. The methodology is reproducible in principle: the Reader is small, all five folds train in 16 minutes, and the fold-assignment file is promised for release. The efficiency argument is attractive, though the 4.6x figure currently overstates the savings because some external perception steps are omitted from the DreamTraj runtime.

major comments (3)
  1. [Sec. 5.4 / App. E.2 / App. B.3] The selection of the readout attention cell (block 28, head 32) and denoising step k=16 is made by a permutation-controlled margin over 955 clips (Eq. 3; Sec. 5.4), with the end-task ADE also used in the step sweep (Table 3, 321 clips). The paper states in Sec. 4.1 that the cell is fixed on training folds, but neither the main text nor the appendix establishes that the 955 clips (or the 321-clip step-sweep subset) are disjoint from the evaluation folds used to produce Tables 1 and 4. If these clips overlap with the held-out folds, the architecture and readout step have effectively been selected on test data, and the reported out-of-fold numbers are optimistically biased. This is load-bearing because App. E.2 shows that only three attention blocks yield usable tracks and the margin ranking is decisive for end-task ADE. I request a clear statement, or a released fold-assignment file, demon
  2. [Table 2 / Sec. 5.3] The efficiency comparison omits two components that DreamTraj actually executes: the object grounding step (GroundingDINO + SAM2, used in Sec. 4.1 and Fig. 2) and the monocular depth estimation that restores metric scale (Sec. 4.2, with reference to Yang et al. 2024). Table 2 reports DreamTraj as no external perception with 76s total, while the generate-then-extract cascade includes its full perception stack. This apples-to-oranges accounting inflates the 4.6x speedup factor. The accuracy comparison is unaffected, but the efficiency claim should be corrected by including the grounding and depth inference times, or by explicitly defining the comparison boundary (e.g., excluding shared one-time per-frame preprocessing).
  3. [Table 1 / Sec. 3.2 / Sec. 4.3] The use of generated videos and their pipeline labels for both training (domain B) and evaluation (Table 1b) raises a mild circularity concern: the method is trained on features of generated videos and evaluated on the same type of generated videos, where the labels come from the same perception cascade used to ground the training data. The paper partially mitigates this by reporting separate real-capture results (Table 1a) and explicitly treating the generated half as a feature-alignment domain. I accept that mitigation, but the manuscript should state more clearly that the generated-domain numbers are not intended as evidence of real-world generalization; the real-capture, real-GT block is the primary evidence. This is a framing issue rather than a correctness error.
minor comments (6)
  1. [Abstract / Contributions] Typo in contributions: 'A object-centric trajectory dataset' should be 'An object-centric trajectory dataset.' Also 'Movedataset' appears with no space in Sec. 1 and other places; unify to 'MOVE dataset.'
  2. [Sec. 3.3 / Fig. 1] The name 'RIGVid' is used in Sec. 2.3 and elsewhere while the baseline is cited as 'Patel et al. 2026'; Fig. 1 and Sec. 3.3 write 'RigVid.' Please use one spelling consistently.
  3. [Table 3] The note that 'its scale is not comparable to Table 1' is insufficiently explained. State explicitly that the ADE in Table 3 is computed on the 321-clip generated-video path only, with a different training regime, and is used solely for relative comparisons across k.
  4. [App. E.3 / Fig. 6] The caption of Fig. 6 mentions 'end-task ADE in panel (b),' but the main text does not reference Fig. 6 and the panel (b) description is vague. Clarify how panel (b) is generated and why its numbers differ from Table 3.
  5. [App. C.2] The description of EgoScaler's 'normalisation leakage' and the resulting fix is helpful, but the statement that 'final-step error falls below average error' is tightly argued; a brief explanation of why this cannot happen for an unbiased forecaster would improve the reproducibility narrative.
  6. [Sec. 5.1 / App. A.3] The Reader training is described as 200 epochs with 'one epoch is 29 optimiser steps at batch size 128,' but 29 steps x 128 = 3,712 samples, which is less than the 5,038-total dataset. This is likely because each fold trains on 80% of the data (~4,030 samples), but the text should state the effective training-set size per fold.

Circularity Check

0 steps flagged

No significant circularity; the central accuracy claims are grounded in real-capture supervision and the generated domain is explicitly treated as feature alignment.

full rationale

The derivation chain is not circular. DreamTraj's readout is a trained flow-matching Reader that maps frozen Wan2.2 q·k attention tracks and pooled hidden states to relative 6-DoF pose tokens. The real-capture half of the MOVE dataset is supervised by external 6-DoF annotations from six source corpora, not by the model's own outputs. The generated half is explicitly labeled by an independent perception pipeline (GroundingDINO/SAM2/FoundationPose etc.) and is described as a feature-alignment domain only, so the paper does not present pipeline labels as physical ground truth. The real-capture, real-GT block of Table 1 provides an independent benchmark on which DreamTraj outperforms the privileged-input baselines, which is sufficient to support the central accuracy claim. The attention-as-tracker premise is based on external prior work (Nam et al., Son et al., Tang et al.) and is further validated by a permutation-controlled margin; neither the q·k cell nor the readout step is defined in terms of the final 6-DoF output. Section 4.1 states the cell is fixed on training folds, which, if accurate, prevents the selection from leaking evaluation information. Minor caveats exist: the provenance of the 955/321 selection clips is not explicitly tied to training folds in Sec. 5.4/App. E.2, and Table 2 omits the cost of the grounding/depth modules used at inference. These affect reproducibility and the exact speedup factor, but they do not make any reported prediction equivalent to its inputs by construction, and no load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The central claim rests primarily on the empirical assumption that a single attention head of a frozen video diffusion model encodes object motion at an early denoising step. That assumption is validated on a subset of clips, but the subset's relation to the evaluation folds is not documented. The method also depends on standard pretrained components (grounding, depth, backbone) and hand-set training hyperparameters.

free parameters (7)
  • Readout attention cell (block, head) = block 28, head 32
    Selected by permutation-controlled margin over 955 clips. The paper says the cell is fixed on training folds but does not show the 955-clip subset is fold-disjoint (Sec 5.4, Eq 3, Appendix E.2).
  • Readout denoising step k = 16 of 40 (t≈882)
    Chosen as the earliest step at which the permutation margin saturates after the expert switch; the same subset concern applies (Table 3).
  • Pooled hidden blocks = {0, 19, 39}
    Selected by comparing {0,20,39} on the end task under cross-validation (Appendix E.4, Table 11).
  • Domain mixing ratio = 60% domain A per batch after 40-epoch warm-up
    Hand-set in Sec. A.3 (Table 7); not ablated.
  • Flow-matching auxiliary loss weights = endpoint 0.1, SO(3) chordal 0.1, smoothness 0.05
    Hand-set in Sec. A.3; the central accuracy numbers are not shown to be robust to these choices.
  • Reader capacity and sampling = width 256, 4 DiT blocks, 8 heads, 20 Euler steps, stride-2 caching
    Architecture and inference choices; not systematically ablated aside from the listed ablations.
  • PCA dimension and anchor grid = 128 dims, 4x4 grid
    Chosen in Sec. 4.1; the paper gives no sensitivity analysis.
axioms (5)
  • domain assumption The frozen Wan2.2 backbone's q·k attention maps at the chosen cell implicitly track the manipulated object across imagined frames.
    This is the core premise of the readout; validated only via the permutation margin on 955 clips (Sec 5.4).
  • domain assumption The object can be grounded in the first frame by GroundingDINO+SAM2, and that localization carries through the latent readout.
    Used in Fig 2 and Sec 4.1; if grounding fails, the query patch is wrong.
  • domain assumption A monocular depth estimate from the input frame provides sufficient metric scale.
    Sec 4.2; the authors note a sensor can substitute, but in the reported experiments a monocular model is used.
  • domain assumption The perception pipeline (FoundationPose, DA3, SAM 3D, SpatialTracker v2) produces sufficiently accurate 6-DoF labels for generated videos.
    Sec 3.3; these labels are used to train and evaluate domain B, though the central SOTA is also shown on real GT.
  • standard math Standard flow-matching and SO(3) differential geometry are used without modification.
    Sec 4.2 and Appendix A.3.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents." pith.science (2026). https://pith.science/paper/XT5C2GF6

@misc{pith2026260800486,
  author       = {Pith},
  title        = {Pith review of: DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XT5C2GF6}},
  note         = {Machine review of arXiv:2608.00486}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.

Figures

Figures reproduced from arXiv: 2608.00486 by Boyu Wang, Feng Zheng, Jinyu Yang, Luyang Xie, Tongsheng Ding, Yixuan Yang, Zhen Luo.

Figure 1
Figure 1. Figure 1: Trajectory extraction pipeline used to label gen￾erated videos (Sec. 3.3). Amber marks the two blocks that depart from RigVid and ObjectForesight: DA3 metric depth and a SAM 3D Objects mesh. 2024a), given the object mesh, metric depth, and camera geometry from SpatialTracker v2 (Xiao et al. 2025). Two changes are necessary for generated content: metric depth from DA3 (Lin et al. 2025) rather than a relativ… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DreamTraj. One RGB frame and one instruction go in. The object is grounded once in that frame (GroundingDINO + SAM2) to give query points; a frozen Wan2.2-I2V-A14B is stopped at step 16 of 40, and from that latent we read q·k attention tracks (block 28, head 32) and anchor-pooled hidden states (blocks 0/19/39) as a 13 × 132 sequence. A 7.1M-parameter flow-matching trainable Reader, turns it int… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison. Each row is one clip, labelled with the instruction the model was given (abridged for space). Panels overlay 7 of the 13 predicted poses on the anchor frame, each outlined in green, with the dashed line tracing the object’s centre. Red marks error: a centroid off by more than the object’s own width, or an orientation off by more than 45◦ . Methods marked † receive privileged input: … view at source ↗
Figure 4
Figure 4. Figure 4: Generalization beyond the training distribution. Scenes, objects and actions absent from every training corpus, rendered as in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Move statistics. (a) source corpus, (b) action verb, (c) object category, each split into recorded and generated trajectories; (d) joint distribution of per-clip translation and rotation magnitude over all 5,038 trajectories. the independent step sweep of [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Readout step sweep. (a) permutation-controlled track margin against the recorded 2-D object track; (b) end-task ADE from retraining the Reader at each step. The shaded region is the backbone’s high-noise expert. The signal appears at the expert switch, not gradually. this is not a coarse early-versus-late effect: the readout is sensitive to where in the stack the scene context is taken from, which is why t… view at source ↗
Figure 7
Figure 7. Figure 7: Extended qualitative comparison. Each row is one clip, labelled with the instruction the model was given (abridged for space). Panels overlay 7 of the 13 predicted poses on the anchor frame, each outlined in green, with the dashed line tracing the object’s centre. Red marks error: a centroid off by more than the object’s own width, or an orientation off by more than 45◦ . Methods marked † receive privilege… view at source ↗
Figure 8
Figure 8. Figure 8: Generalization beyond the training distribution. Scenes, objects and actions absent from every training corpus, rendered as in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 38 canonical work pages · 2 internal anchors

  1. [1]

    J.; and Hodan, T

    Banerjee, P.; Shkodrani, S.; Moulon, P.; Hampali, S.; Han, S.; Zhang, F.; Zhang, L.; Fountain, J.; Miller, E.; Basol, S.; Newcombe, R.; Wang, R.; Engel, J. J.; and Hodan, T. 2025. HOT3D : Hand and Object Tracking in 3D from Egocentric Multi-View Videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2025

  2. [2]

    Bharadhwaj, H.; Dwibedi, D.; Gupta, A.; Tulsiani, S.; Doersch, C.; Xiao, T.; Shah, D.; Xia, F.; Sadigh, D.; and Kirmani, S. 2024 a . Gen2Act: Human Video Generation in Novel Scenarios Enables Generalizable Robot Manipulation. In Conference on Robot Learning

    work page 2024

  3. [3]

    Bharadhwaj, H.; Mottaghi, R.; Gupta, A.; and Tulsiani, S. 2024 b . Track2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation. In European Conference on Computer Vision

    work page 2024

  4. [4]

    Damen, D.; et al. 2018. Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. In European Conference on Computer Vision

    work page 2018

  5. [5]

    Dharmarajan, K.; Huang, W.; Wu, J.; Fei-Fei, L.; and Zhang, R. 2025. Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow. ArXiv:2512.24766

    arXiv 2025

  6. [6]

    B.; Schuurmans, D.; and Abbeel, P

    Du, Y.; Yang, S.; Dai, B.; Dai, H.; Nachum, O.; Tenenbaum, J. B.; Schuurmans, D.; and Abbeel, P. 2023. Learning Universal Policies via Text-Guided Video Generation. In Advances in Neural Information Processing Systems

    work page 2023

  7. [7]

    J.; and Hilliges, O

    Fan, Z.; Taheri, O.; Tzionas, D.; Kocabas, M.; Kaufmann, M.; Black, M. J.; and Hilliges, O. 2023. ARCTIC : A Dataset for Dexterous Bimanual Hand-Object Manipulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2023

  8. [8]

    Garcia-Hernando, G.; Yuan, S.; Baek, S.; and Kim, T.-K. 2018. First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2018

  9. [9]

    Grauman, K.; et al. 2022. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2022

  10. [10]

    Grauman, K.; et al. 2024. Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2024

  11. [11]

    Huang, H.; Yuan, X.; Liao, P.; Teng, T.; Yang, C.; Guo, J.; and Guo, Y. 2026. GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency. ArXiv:2607.09191

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [12]

    P.; Ye, J

    Jeong, H.; Huang, C.-H. P.; Ye, J. C.; Mitra, N. J.; and Ceylan, D. 2025. Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2025

  13. [13]

    Khazatsky, A.; et al. 2024. DROID : A Large-Scale In-the-Wild Robot Manipulation Dataset. In Robotics: Science and Systems

    work page 2024

  14. [14]

    Ko, P.-C.; Mao, J.; Du, Y.; Sun, S.-H.; and Tenenbaum, J. B. 2024. Learning to Act from Actionless Videos through Dense Correspondences. In International Conference on Learning Representations

    work page 2024

  15. [15]

    Kwon, T.; Tekin, B.; St \"u hmer, J.; Bogo, F.; and Pollefeys, M. 2021. H2O : Two Hands Manipulating Objects for First Person Interaction Recognition. In IEEE/CVF International Conference on Computer Vision

    work page 2021

  16. [16]

    Li, H.; Sun, L.; Hu, Y.; Ta, D.; Barry, J.; Konidaris, G.; and Fu, J. 2025. NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos. ArXiv:2510.08568

    arXiv 2025

  17. [17]

    Li, Y.; Zhou, X.; Ge, Y.; and Kong, Y. 2026. Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation. ArXiv:2605.20085

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  18. [18]

    Liang, J.; Liu, R.; Ozguroglu, E.; Sudhakar, S.; Dave, A.; Tokmakov, P.; Song, S.; and Vondrick, C. 2024. Dreamitate: Real-World Visuomotor Policy Learning via Video Generation. In Conference on Robot Learning

    work page 2024

  19. [19]

    Y.; Li, Z.; Shi, G.; Feng, J.; and Kang, B

    Lin, H.; Chen, S.; Liew, J.; Chen, D. Y.; Li, Z.; Shi, G.; Feng, J.; and Kang, B. 2025. Depth Anything 3: Recovering the Visual Space from Any Views. ArXiv:2511.10647

    Pith/arXiv arXiv 2025

  20. [20]

    Lipman, Y.; Chen, R. T. Q.; Ben-Hamu, H.; Nickel, M.; and Le, M. 2023. Flow Matching for Generative Modeling. In International Conference on Learning Representations

    work page 2023

  21. [21]

    Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Jiang, Q.; Li, C.; Yang, J.; Su, H.; Zhu, J.; and Zhang, L. 2024 a . Grounding DINO : Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. In European Conference on Computer Vision

    work page 2024

  22. [22]

    Liu, Y.; Liu, Y.; Jiang, C.; Lyu, K.; Wan, W.; Shen, H.; Liang, B.; Fu, Z.; Wang, H.; and Yi, L. 2022. HOI4D : A 4D Egocentric Dataset for Category-Level Human-Object Interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2022

  23. [23]

    Liu, Y.; Yang, H.; Si, X.; Liu, L.; Li, Z.; Zhang, Y.; Liu, Y.; and Yi, L. 2024 b . TACO : Benchmarking Generalizable Bimanual Tool-Action-Object Understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2024

  24. [24]

    Nam, J.; Son, S.; Chung, D.; Kim, J.; Jin, S.; Hur, J.; and Kim, S. 2025. Emergent Temporal Correspondences from Video Diffusion Transformers. In Advances in Neural Information Processing Systems

    work page 2025

  25. [25]

    Open X-Embodiment Collaboration . 2024. Open X-Embodiment : Robotic Learning Datasets and RT-X Models. In IEEE International Conference on Robotics and Automation

    work page 2024

  26. [26]

    Patel, S.; Mohan, S.; Mai, H.; Jain, U.; Lazebnik, S.; and Li, Y. 2026. Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations. In International Conference on Learning Representations

    work page 2026

  27. [27]

    Peebles, W.; and Xie, S. 2023. Scalable Diffusion Models with Transformers. In IEEE/CVF International Conference on Computer Vision

    work page 2023

  28. [28]

    Qwen Team . 2026. Qwen3.5 . Model release

    work page 2026

  29. [29]

    Ravi, N.; et al. 2024. SAM 2 : Segment Anything in Images and Videos. ArXiv:2408.00714

    Pith/arXiv arXiv 2024

  30. [30]

    SAM 3D Team . 2025. SAM 3D : 3D fy Anything in Images. ArXiv:2511.16624

    Pith/arXiv arXiv 2025

  31. [31]

    Saroha, A.; Zeng, H.; Zuo, X.; Cremers, D.; and Wang, X. 2026. EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation. ArXiv:2604.01421

    work page arXiv 2026

  32. [32]

    Shrivastava, A.; Mehta, S.; Geng, D.; and Owens, A. 2026. Point Prompting: Counterfactual Tracking with Video Diffusion Models. In International Conference on Learning Representations

    work page 2026

  33. [33]

    Son, S.; An, H.; Nam, J.; Ko, H.; Kim, C.; Chung, D.; Jin, S.; Yi, J.; Hur, J.; and Kim, S. 2025. Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking. ArXiv:2512.20606

    Pith/arXiv arXiv 2025

  34. [34]

    Soraki, R.; Bharadhwaj, H.; Farhadi, A.; and Mottaghi, R. 2026. ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos. ArXiv:2601.05237

    arXiv 2026

  35. [35]

    P.; and Hariharan, B

    Tang, L.; Jia, M.; Wang, Q.; Phoo, C. P.; and Hariharan, B. 2023. Emergent Correspondence from Image Diffusion. In Advances in Neural Information Processing Systems

    work page 2023

  36. [36]

    Wan Team . 2025. Wan: Open and Advanced Large-Scale Video Generative Models. ArXiv:2503.20314

    Pith/arXiv arXiv 2025

  37. [37]

    Wen, B.; Yang, W.; Kautz, J.; and Birchfield, S. 2024 a . FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2024

  38. [38]

    Wen, C.; Lin, X.; So, J.; Chen, K.; Dou, Q.; Gao, Y.; and Abbeel, P. 2024 b . Any-Point Trajectory Modeling for Policy Learning. In Robotics: Science and Systems

    work page 2024

  39. [39]

    Xiao, Y.; Wang, J.; Xue, N.; Karaev, N.; Makarov, Y.; Kang, B.; Zhu, X.; Bao, H.; Shen, Y.; and Zhou, X. 2025. SpatialTrackerV2 : Advancing 3D Point Tracking with Explicit Camera Motion. In IEEE/CVF International Conference on Computer Vision

    work page 2025

  40. [40]

    Xu, M.; Xu, Z.; Xu, Y.; Chi, C.; Wetzstein, G.; Veloso, M.; and Song, S. 2024. Flow as the Cross-Domain Manipulation Interface. In Conference on Robot Learning

    work page 2024

  41. [41]

    Yang, L.; Kang, B.; Huang, Z.; Zhao, Z.; Xu, X.; Feng, J.; and Zhao, H. 2024. Depth Anything V2 . In Advances in Neural Information Processing Systems

    work page 2024

  42. [42]

    Yoshida, T.; Kurita, S.; Nishimura, T.; and Mori, S. 2025. Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2025

  43. [43]

    Yuan, C.; Wen, C.; Zhang, T.; and Gao, Y. 2024. General Flow as Foundation Affordance for Scalable Robot Learning. In Conference on Robot Learning

    work page 2024

  44. [44]

    Zhan, X.; Yang, L.; Zhao, Y.; Mao, K.; Xu, H.; Lin, Z.; Li, K.; and Lu, C. 2024. OakInk2 : A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2024

  45. [45]

    Zhang, J.; Herrmann, C.; Hur, J.; Polania Cabrera, L.; Jampani, V.; Sun, D.; and Yang, M.-H. 2023. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. In Advances in Neural Information Processing Systems

    work page 2023

  46. [46]

    Zhou, Y.; Barnes, C.; Lu, J.; Yang, J.; and Li, H. 2019. On the Continuity of Rotation Representations in Neural Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2019

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.