Pith. sign in

REVIEW 4 major objections 8 minor 10 cited by

Trajectory Attention for Fine-grained Video Motion Control

T0 review · 4 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding an auxiliary attention branch that matches features along camera-defined pixel trajectories gives pretrained video diffusion models precise, long-range motion control without retraining the base model.

desk verdict Genuinely plausible new attention mechanism for camera control, but the reported precision gains rest on an unstated trajectory-estimation protocol that needs to be pinned down before the numbers can be believed. read the letter →

arxiv 2411.19324 v1 pith:YWZPTGL4 submitted 2024-11-28 cs.CV

Zeqi Xiao , Wenqi Ouyang , Yifan Zhou , Shuai Yang , Lei Yang , Jianlou Si , Xingang Pan This is my paper · ORCID
classification cs.CV
keywords trajectoryattentioncameramotioncontrolvideodiffusionmodelstemporalgenerationfirst-frame-guidededitingmechanism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that camera motion in video generation can be controlled precisely by attending along the pixel trajectories the camera would trace, rather than by feeding pose embeddings or warped frames into the model. The key idea is to keep the ordinary temporal attention untouched and add a second, auxiliary attention branch that samples hidden features at trajectory coordinates, runs attention with its own parameters, and adds the result back as a residual. The authors argue that temporal attention mostly locks onto nearby frames, so it cannot enforce long-range consistency, while trajectory attention is free to align features over the whole path and still lets the model generate new content where no trajectory is given. If the claim holds, the same training-light recipe should transfer across base models and tasks, which the paper supports with image camera control, video camera control, and first-frame-guided video editing experiments.

What carries the argument

Trajectory attention is an auxiliary multi-head attention module that operates on features sampled along user-specified pixel trajectories instead of on the full spatial grid. For each trajectory, Algorithm 1 reads the hidden states at the trajectory's 2D coordinates in every frame, masks invalid frames, and stacks them into a sequence; after attention, Algorithm 2 scatters the updated features back to their original locations and normalizes by how many trajectories covered each pixel. The QKV projectors are copied from the temporal-attention module, while the output projection is zero-initialized, so the branch starts as a no-op and grows gradually during fine-tuning. It is this sampled-along-trajectory, read-and-scatter attention that injects the camera motion as an explicit correspondence constraint while leaving the base model's parameters untouched.

What would settle it

On a benchmark with known camera poses and ground-truth depth, generate the same scenes twice, once with trajectories computed from ground-truth depth and once from the paper's estimated depth with its default camera assumptions, then compare ATE and RPE. If pose error does not increase when the depth map is corrupted or replaced, trajectory accuracy is not what drives the control; if error rises sharply, the dependence on trajectory extraction is confirmed. A complementary test is to ablate the auxiliary branch itself by feeding the sampled trajectory features directly into the temporal attention's value stream; if control precision stays equally high, the auxiliary-branch design is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that a dedicated trajectory-attention branch, initialized from the temporal-attention weights and added to it as a zero-initialized residual, is sufficient to make a pretrained video diffusion model follow a specified camera path with much lower error than pose-embedding or warping-based conditioning. Concretely, hidden states are sampled at each frame along the supplied 2D trajectories, attended to frame-wise, and written back and averaged at the sampled locations, so the attention mechanism itself carries the motion condition. The paper reports that this design reduces absolute trajectory error and relative pose error on single-image camera control (for example, an ATE of 0.0396 on the 25-frame setting versus 0.0411 for the best pose-conditioning baseline) and on video camera control, while keeping FID comparable. It also shows the branch extends from 12-frame training to 25-frame generation and to full 3D-attention models, and that it can drive first-frame-guided video editing. The role of the auxiliary branch is to enforce long-range consistency along the trajectory without forcing the original temporal attention to abandon its short-range motion synthesis.

Load-bearing premise

The load-bearing premise is that the extracted pixel trajectories are accurate: they come from an estimated depth map, fixed assumptions about the camera's focal length and center, and, for videos, a point tracker, so any error in those estimates means the attention aligns features along paths that are not the true 3D trajectories.

Editorial extensions

If this is right

  • Single-image camera control becomes a fine-grained operation: reported absolute trajectory error drops to roughly 0.02-0.04 meters on the tested 14-, 16-, and 25-frame settings, close to the ground-truth camera path.
  • Because trajectory attention is a residual add-on, it combines with orthogonal techniques such as warping-based test-time optimization, improving both precision and fidelity over either approach alone.
  • The same branch transfers from 12-frame training to 25-frame generation, suggesting the learned trajectory prior is not tied to a specific sequence length.
  • Trajectory attention extends beyond camera motion to first-frame-guided video editing, preserving edited content across large spatial and temporal ranges where frame-warping methods drift.
  • The design adapts to full 3D-attention video architectures, so it is not limited to decomposed spatial-temporal attention models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same auxiliary-branch recipe could apply to other correspondence-based controls, such as object dragging, depth-guided motion, or user-sketched flow, since the branch never needs to know where the trajectories came from.
  • Because the fixed camera intrinsics are acknowledged as a source of error, a natural next step is to jointly predict or calibrate intrinsics from the image, which would likely extend the method to 360-degree and wide-angle camera paths.
  • The attention-window visualization suggests a quantitative diagnostic: tracking the effective window width of the trajectory branch during generation should correlate with long-range consistency, giving a cheap way to predict when control will fail on fast motion.
  • Since the branch is trained on only 10k clips, the method appears data-efficient, but the paper does not show how performance scales with much larger trajectory-conditioned training sets, so the ceiling of the approach remains unknown.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces trajectory attention, an auxiliary attention branch that operates on features sampled along pixel trajectories, in parallel with the standard temporal attention of a video diffusion model. The branch is initialized from the temporal attention weights, its output is added as residuals, and only this branch is fine-tuned. The method is applied to camera motion control from a single image, camera motion control on videos, and first-frame-guided video editing. The experiments report ATE/RPE/FID improvements over several baselines, and the paper demonstrates the extension to full 3D attention models. The authors also provide ablations on the branch design, training data, and trajectory sparsity.

Significance. The core idea is elegant and practically valuable: rather than replacing or modifying temporal attention, trajectory attention is added as a lightweight, plug-and-play branch with weight inheritance and zero-initialized output, enabling efficient fine-tuning (about 24 GPU hours) without changing the base model parameters. The attention-map analysis (Fig. 2) provides a sensible motivation for the auxiliary-branch design. If the reported precision gains are reproducible, the method would be a significant step for controllable video synthesis. The paper is also transparent about its limitations (Appendix A.1) regarding dependence on external trajectory extractors and failure modes on fast or complex motions. However, the evaluation protocol has gaps that currently prevent full verification of the central precision claim.

major comments (4)
  1. [Sec. 5.1, Tables 1-2] The paper reports ATE and RPE as the headline metrics for camera-control precision, but it never states how camera trajectories are recovered from the generated videos. Appendix A.9 shows 'estimated trajectories from our generation' without naming the estimator (COLMAP, monocular SLAM, or a learned method) or the alignment procedure. This is load-bearing: if the estimator is sensitive to video smoothness or temporal consistency, a method that produces more static videos could appear to have lower pose error even when it follows the commanded camera motion less accurately. Please specify the full evaluation protocol, including the estimator, alignment (Sim(3)/SE(3)), and any outlier handling, and provide a sensitivity analysis.
  2. [Sec. 5.5, Table 3] The ablation states that for the 'Vanilla' condition, 'some outputs exhibiting complete noise (we omit such invalid results during evaluation).' This selection biases the reported ATE/RPE/FID and makes the comparison with the proposed design difficult to interpret. Please report the fraction of invalid outputs for each ablation condition, and either include a failure-rate metric (e.g., percentage of runs that produce non-finite or degenerate outputs) or use a metric that handles failures. Without this, the large gap between Vanilla and the proposed design is not fully supported.
  3. [Sec. 5.1 and Table 1] The quantitative comparison mixes base models and frame lengths: MotionI2V uses AnimateDiff while the other baselines and the proposed method use SVD, and the comparisons are at 14, 16, and 25 frames. The table footnote acknowledges this, but the text claims 'consistently achieve higher or comparable control precision' across settings. To attribute the improvements to the trajectory-attention design, please provide a same-base-model, same-frame-length comparison at least for the strongest baseline, or clearly state the confound. The 10-100x ATE improvements over MotionCtrl may partly reflect the different base models or frame lengths.
  4. [Appendix A.3 and Sec. 4.1] Single-image camera control uses predefined camera intrinsics (fx=fy=260, principal point at image center) and a depth estimate from DepthAnythingV2 to compute the conditioning trajectories. The paper does not quantify how the reported ATE/RPE depend on these choices, and it is unclear whether the same depth/projection pipeline is used to extract the 'estimated trajectories' from the generated videos that enter the ATE/RPE computation. If the same pipeline is used, the evaluation is partially circular. Please clarify the evaluation trajectory extraction and provide a quantitative sensitivity analysis over focal lengths (or other intrinsic parameters), beyond the qualitative illustration in Fig. 21.
minor comments (8)
  1. [Algorithm 1] Algorithm 1 does not specify how fractional 2D coordinates are sampled from the hidden states; please state whether nearest-neighbor or bilinear (or other) interpolation is used.
  2. [Sec. 3.4] Equation (3) does not define the conditioning signal c or the noise schedule σ; for reproducibility, please specify what c is for each task (e.g., trajectories and first frame) and cite the schedule used by SVD.
  3. [Sec. 5.1 and A.10] The citation 'Yang et al. (2023a)' for optical flow extraction points to a paper on semi-supervised semantic segmentation; the intended optical-flow reference appears to be missing or mis-cited. Please correct.
  4. [Tables 1 and 2] Tables 1 and 2 are titled 'Qualitative comparison' but contain quantitative metrics; please rename them 'Quantitative comparison' or similar.
  5. [Algorithm 4] In Algorithm 4, the combination Tr = Pt + P assumes that Pt and P are in the same pixel-coordinate frame; please state how the sampled camera translation Pt and the point trajectories P are aligned, and what happens when a point becomes occluded.
  6. [Sec. 5 and throughout] The paper claims 'long-range consistency' but no temporal consistency metric (e.g., per-video LPIPS or feature stability) is reported; the FID is a distribution-level metric and does not directly measure temporal consistency within a generated clip.
  7. [Appendix A.9] Figure 16 shows estimated trajectories from 'our generation' and from CameraCtrl, but the estimation method is not described; please name the estimator and state whether it is the same for all compared methods.
  8. [Various] Minor typos: 'straighwarpward' (Sec. 3.2), 'lenght' (Algorithm 4), 'trajecories' (Algorithms 3 and 4), and 'wrapped frames' (Fig. 13) should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central precision claim is tested against external ATE/RPE benchmarks and the trajectory-attention design is a genuine architectural contribution.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The conditioning signal (trajectories) is extracted by explicitly stated geometric procedures (Algorithms 3 and 4, Appendix A.3) from depth estimates, hand-set intrinsics, and camera poses, while the training loss (Eq. 3) is the standard denoising objective and contains no ATE/RPE term. The claimed improvements are evaluated on ATE and RPE computed against ground-truth camera trajectories, which are not part of the network's conditioning input at inference time. The key design choices—auxiliary branch, parameter sharing, and zero-initialized output projection—are architectural decisions validated by ablations (Table 3), not fitted constants renamed as predictions. The paper's self-references (e.g., Ouyang et al. 2024, Xiao et al. 2024) appear only as related-work context and are not load-bearing. The main caveat is that Section 5.1 never specifies how camera trajectories are estimated from the generated videos, and Appendix A.9 only shows qualitative trajectory alignments; however, this is an evaluation-protocol gap and a correctness risk, not a circular reduction, because no equation or algorithm in the paper makes the reported ATE/RPE values equal to the conditioning trajectories by construction. Appendix A.1's acknowledged dependence on external trajectory estimators is a stated limitation rather than evidence that the central claim reduces to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; its central claim rests on learned modules and external trajectory estimates. The main hand-set input is the assumed camera intrinsics for single-image control, which directly shapes every reported image-control number. The design assumptions about weight transfer and branch synergy are empirically validated rather than derived.

free parameters (2)
  • Predefined camera intrinsics for single-image control = fx=fy=260, cx=W/2, cy=H/2
    Chosen by hand in Appendix A.3 because intrinsics cannot be estimated from a single image; all image camera-control evaluations depend on trajectories derived from these values.
  • Number of sampled trajectories L and density = Unspecified for main experiments; appendix reports robustness at 1/16 and instability below 1/32 resolution
    The amount of trajectory guidance is a critical design choice affecting control precision; the main paper does not report the density used in the reported evaluations.
assumptions (3)
  • domain assumption Pre-trained temporal attention in SVD has learned adequate motion priors that transfer to trajectory attention via weight initialization.
    Sec. 3.4 and Fig. 5 copy QKV weights from temporal attention; the method's gains depend on this transfer working.
  • domain assumption Depth estimation (DepthAnythingV2) and point tracking (CoTracker) produce trajectories accurate enough for attention sampling.
    Algorithms 3-4 feed these estimates as conditions; the paper's Appendix A.1 lists failures on fast and complex motion.
  • ad hoc to paper Adding trajectory attention as a residual branch does not interfere with temporal attention's synthesis.
    Sec. 3.3 asserts the two branches have distinct purposes and 'work in synergy'; this is validated only empirically and is the core design hypothesis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trajectory Attention for Fine-grained Video Motion Control." pith.science (2026). https://pith.science/paper/YWZPTGL4

@misc{pith2026241119324,
  author       = {Pith},
  title        = {Pith review of: Trajectory Attention for Fine-grained Video Motion Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YWZPTGL4}},
  note         = {Machine review of arXiv:2411.19324}
}
read the original abstract

Recent advancements in video generation have been greatly driven by video diffusion models, with camera motion control emerging as a crucial challenge in creating view-customized visual content. This paper introduces trajectory attention, a novel approach that performs attention along available pixel trajectories for fine-grained camera motion control. Unlike existing methods that often yield imprecise outputs or neglect temporal correlations, our approach possesses a stronger inductive bias that seamlessly injects trajectory information into the video generation process. Importantly, our approach models trajectory attention as an auxiliary branch alongside traditional temporal attention. This design enables the original temporal attention and the trajectory attention to work in synergy, ensuring both precise motion control and new content generation capability, which is critical when the trajectory is only partially available. Experiments on camera motion control for images and videos demonstrate significant improvements in precision and long-range consistency while maintaining high-quality generation. Furthermore, we show that our approach can be extended to other video motion control tasks, such as first-frame-guided video editing, where it excels in maintaining content consistency over large spatial and temporal ranges.

Figures

Figures reproduced from arXiv: 2411.19324 by the authors.

Figure 1
Figure 1. Trajectory attention injects partial motion information by making content along trajecto￾ries consistent. It facilitates various tasks such as camera motion control on images and videos, and first-frame-guided video editing. Yellow boxes indicate reference contents. Green boxes indicate input frames. Blue boxes indicate output frames. In response to these limitations, recent methods have begun to address temporal re… view at source ↗
Figure 2
Figure 2. Attention map visualization of temporal attention and trajectory attention. (a) Tem￾poral attention tends to concentrate its weight on a narrow, adjacent frame window. (b) In contrast, trajectory attention exhibits a broader attention window, highlighting its capacity to produce more consistent and controllable results. Here, the attention map is structured with the frame number as the side length. The attention wei… view at source ↗
Figure 3
Figure 3. Overview of the proposed motion control pipeline. Our method allows for conditioning on trajectories from various sources – such as camera motion derived from a single image, as shown in this figure. We inject these conditions into the model through trajectory attention, enabling explicit and fine-grained control over the motion in the generated video. number of frames, height, width, and channels, respectively, tem… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Visualization of vanilla tem￾poral attention and trajectory atten￾tion. In contrast, trajectory attention, given its known dynam￾ics, aims solely to align features along the trajectory. This singular focus on alignment often clashes with the broader objectives of tempo…
Figure 5
Figure 5. Figure 5: Training strategy for trajectory attention. To leverage the motion modeling capability learned from large-scale data, we initialize the weights of the QKV projectors with those from temporal attention layers. Additionally, the output projector is initialized with zero …
Figure 6
Figure 6. Figure 6: Qualitative comparisons for camera motion control on images. While other meth￾ods often exhibit significant quality degradation or inconsistencies in camera motion, our approach consistently delivers high-quality results with precise, fine-grained control over camera m…
Figure 7
Figure 7. Figure 7: Qualitative comparisons for camera motion control on videos. In the second row, we provide video frames after view warping as a reference. Methods like NVS Solver (You et al., 2024) use frame-wise information injection but overlook temporal continuity, leading to incon…
Figure 8
Figure 8. Figure 8: Results on first-frame guided video editing. We compare our method with those from Ouyang et al. (2024); Ku et al. (2024). The results show that other methods struggle to maintain consistency after editing. In contrast, our method successfully preserves the edited feat…
Figure 9
Figure 9. Figure 9: Qualitative results on Open-Sora-Plan.(Lab & etc., 2024) By incorporating trajectory attention into the 3D attention module, we successfully enable camera motion control. 5.6 RESULTS ON FULL ATTENTION MODELS. Our method also has the potential to support full 3D attenti…
Figure 10
Figure 10. Figure 10: Pipe for video diffusion models with 3D Attention. The key distinction with the pipeline in the main paper lies in applying trajectory attention to the 3D attention module, rather than to the temporal attention mechanism. A.3 DETAILS OF TASK PROCESS Camera Parameters.…
Figure 11
Figure 11. Figure 11: Depth estimation results. Translation computing. Based on the depth map, camera parameters of two views, we can get the translation of pixels shown in Alg. 5. Algorithm 5: Compute pixel translation Input: D ∈ R Hp×Wp : Depth map of the first view E1 ∈ R 4×4 : Extrinsi…
Figure 12
Figure 12. Figure 12: Point trajectory estimation results [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Input process visualization. For all tasks, the inputs to the network are the first frame and the extracted trajectories. The usage of the first frame and the trajectories are identical to [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Using synthetic optical flow as guidance. Our method supports directly using optical flow to guide generation. Blue boxes indicate the optical flow. Yellow boxes indicate the reference image. A.7 MORE CHALLENGING CASES. As shown in [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 15
Figure 15. Figure 15: Examples of challenging situations. Our method effectively addresses complex sce￾narios, including (a) video editing involving multiple objects, (b) video editing in the presence of occlusions, (c) diverse and rapid camera movements, such as zooming in and out, as wel…
Figure 16
Figure 16. Figure 16: Visualization of camera trajectories. The first row displays the estimated trajectories from our generation alongside the ground truth trajectories. The second row presents the estimated trajectories from CameraCtrl (denoted as “CC”) compared to the ground truth. The …
Figure 17
Figure 17. Figure 17: Results on sparse trajectories. In (a), we show that trajectory attention remains robust even with relatively sparse trajectories. Even when the trajectory density is reduced to 1/16 of the original video resolution, it still performs well in motion control. In (b), w…
Figure 18
Figure 18. Figure 18: Applications on drag signals. Trajectory attention supports hand-crafted dragging tra￾jectory. Row 1: origin videos. Row 2: dragged results. Frame 1 Frame 1 Frame Fwd Flow Fwd Occ. Mask Frame Fwd Flow Fwd Occ. Mask [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Visualization on the training data. The training data includes origin frames, predicted optical flow, and occlusion masks. (a) Failure case due to high speed camera motion (b) Failure case due to complex motion and poor trajectory estimation Reconstructed results Orig…
Figure 20
Figure 20. Figure 20: Visualization on failure cases. Our method encounters challenges when dealing with extremely fast motions as well as complex and difficult-to-estimate motion patterns. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21 [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    TryOnCrafter is the first DiT-based framework for camera-controllable video virtual try-on via a renderable 4D try-on proxy distilled from 2D priors into 3DGS avatar animated with SMPL-X.

  2. Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation

    cs.CV 2025-01 conditional novelty 7.0 of 10

    A diffusion-based image animation method that uses rendered colored spheres and a checkerboard world envelope as 3D-aware motion control signals, enabling joint camera and object motion control with fine granularity.

  3. Wonderland: Navigating 3D Scenes from a Single Image

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A feed-forward pipeline reconstructs 3D Gaussian scenes from single images by regressing 3DGS directly from camera-conditioned video diffusion latents.

  4. InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A training-free, near-zero-overhead inverse solver for novel-view video generation and inpainting that projects masks into continuous multi-channel latent masks and applies DDS with conjugate gradient in latent space.

  5. PostCam: Camera-Controllable Novel-View Video Generation with Query-Shared Cross-Attention

    cs.CV 2025-11 conditional novelty 6.0 of 10

    PostCam generates new videos from a reference video along user-specified camera trajectories using a query-shared cross-attention that fuses pose data and rendered frames, improving control precision and detail preservation.

  6. EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    EPiC trains a 30M-parameter visibility-aware ControlNet on mask-based anchor videos from 5,000 in-the-wild videos and 500 steps, reaching SOTA camera accuracy on RealEstate10K and MiraData.

  7. VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    VidCRAFT3 is a single image-to-video diffusion system that accepts camera, object, and lighting direction controls separately or jointly, trained in three stages with a new synthetic lighting dataset.

  8. Follow-Your-Creation: Empowering 4D Creation through Video Inpainting

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Follow-Your-Creation fine-tunes the Wan2.1 video inpainting model on composite point-cloud and editing masks so a single monocular video can be converted into editable 4D video with new camera motion.

  9. Motion Prompting: Controlling Video Generation with Motion Trajectories

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A single-stage ControlNet on the Lumiere video model, conditioned only on dense point tracks, generalizes to sparse and dense trajectory control for object, camera, and transferred motions.

  10. Dynamic View Synthesis as an Inverse Problem

    cs.CV 2025-06 reject novelty 3.0 of 10

    Dynamic view synthesis from a monocular video is achieved by redesigning the noise initialization of a pretrained video diffusion model using a recursive interpolation and a stochastic latent modulation.

Reference graph

Works this paper leans on

39 extracted references · 6 canonical work pages · cited by 10 Pith papers

  1. [1]

    Vd3d: Taming large video diffusion transformers for 3d camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407.12781,

    arXiv

  2. [4]

    Motion- conditioned diffusion model for controllable video synthesis

    Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304.14404, 2023b. Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion m...

    arXiv

  3. [5]

    Dragvideo: Interactive drag-style video editing

    Yufan Deng, Ruida Wang, Yuhao Zhang, Yu-Wing Tai, and Chi-Keung Tang. Dragvideo: Interactive drag-style video editing. arXiv preprint arXiv:2312.02216,

    arXiv

  4. [6]

    Motion guidance: Diffusion-based image editing with differen- tiable motion estimators

    Daniel Geng and Andrew Owens. Motion guidance: Diffusion-based image editing with differen- tiable motion estimators. arXiv preprint arXiv:2401.18085,

    arXiv

  5. [7]

    Robust localization using relative and absolute position estimates

    Puneet Goel, Stergios I Roumeliotis, and Gaurav S Sukhatme. Robust localization using relative and absolute position estimates. In Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No. 99CH36289), volume 2, pp. 1134–1140. IEEE,

    work page 1999

  6. [8]

    Liveportrait: Efficient portrait animation with stitching and retargeting control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168,

    arXiv

  7. [9]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933, 2023a. Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning...

    arXiv

  8. [11]

    Training-free camera control for video generation

    Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation. arXiv preprint arXiv:2406.10126,

    arXiv

Show all 39 references
  1. [12]

    Miradata: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358,

    arXiv

  2. [13]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635,

    arXiv

  3. [14]

    Anyv2v: A plug-and-play frame- work for any video-to-video editing tasks

    Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play frame- work for any video-to-video editing tasks. arXiv preprint arXiv:2403.14468,

    arXiv

  4. [15]

    Collaborative video diffusion: Consistent multi-video generation with camera control

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. arXiv preprint arXiv:2405.17414,

    arXiv

  5. [16]

    5281/zenodo.10948109

    URL https://doi.org/10. 5281/zenodo.10948109. Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin, Mengfei Li, Xingqun Qi, Shanghang Zhang, Wenhan Luo, Ping Tan, et al. Era3d: High-resolution multiview diffusion using efficient row-wise attention. arXiv preprint arXiv:2405.11616,

    arXiv

  6. [17]

    Motionclone: Training-free motion cloning for controllable video generation

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05338,

    arXiv

  7. [18]

    Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to- video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to- video diffusion model. arXiv preprint arXiv:2405.20222,

    arXiv

  8. [19]

    I2vedit: First-frame-guided video editing via image-to-video diffusion models

    Wenqi Ouyang, Yi Dong, Lei Yang, Jianlou Si, and Xingang Pan. I2vedit: First-frame-guided video editing via image-to-video diffusion models. arXiv preprint arXiv:2405.16537,

    arXiv

  9. [20]

    Freetraj: Tuning-free trajectory control in video diffusion models

    Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free trajectory control in video diffusion models. arXiv preprint arXiv:2406.16863,

    arXiv

  10. [21]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. InACM SIGGRAPH 2024 Conference Papers, pp. 1–11,

    2024

  11. [22]

    Drag-a-video: Non-rigid video editing with point-based interaction

    13 Yao Teng, Enze Xie, Yue Wu, Haoyu Han, Zhenguo Li, and Xihui Liu. Drag-a-video: Non-rigid video editing with point-based interaction. arXiv preprint arXiv:2312.02936,

    arXiv

  12. [23]

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li

    URL https://arxiv.org/abs/2408.14837. Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566, 2024a. Jiuniu Wang, Hangjie Yuan, Dayou Chen...

    2024 arXiv

  13. [24]

    Video diffusion models are training-free motion interpreter and controller

    Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint arXiv:2405.14864,

    arXiv

  14. [25]

    Camco: Camera-controllable 3d-consistent image-to-video generation

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509,

    arXiv

  15. [26]

    Revisiting weak-to-strong con- sistency in semi-supervised semantic segmentation

    Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong con- sistency in semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7236–7246, 2023a. Lihe Yang, Bingyi Kang, ...

    2023 arXiv

  16. [27]

    Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089,

    arXiv

  17. [28]

    Nvs-solver: Video diffusion model as zero-shot novel view synthesizer

    Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer. arXiv preprint arXiv:2405.15364,

    arXiv

  18. [29]

    Animatezero: Video diffusion models are zero-shot image animators

    14 Jiwen Yu, Xiaodong Cun, Chenyang Qi, Yong Zhang, Xintao Wang, Ying Shan, and Jian Zhang. Animatezero: Video diffusion models are zero-shot image animators. arXiv preprint arXiv:2312.03793,

    arXiv

  19. [30]

    Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis

    Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409.02048,

    arXiv

  20. [31]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847, 2023a. Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, ...

    arXiv

  21. [32]

    Motiondirector: Motion customization of text-to-video diffusion models

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465,

    arXiv

  22. [33]

    Compositional 3d- aware video generation with llm director

    Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, and Jiang Bian. Compositional 3d- aware video generation with llm director. Advances in Neural Information Processing Systems, 2024a. Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, ...

    arXiv

  23. [34]

    15 A A PPENDIX A.1 L IMITATIONS As shown in Fig

    URL https://arxiv.org/abs/2405.04496. 15 A A PPENDIX A.1 L IMITATIONS As shown in Fig. 20, our method has several limitations that require further exploration. Currently, it depends on external techniques such as Karaev et al. (2023); Yang et al. (2023a) for trajectory extract...

    2023 arXiv

  24. [35]

    model with 3D attention. The trajectory attention is constructed and trained following the same procedure outlined in the main paper, with the key difference being that it is appended to the 3D full attention block instead of the temporal attention block (Fig. 10). For other t...

    2024

  25. [36]

    We use CoTracker Karaev et al

    7 Pworld ← Dreshaped · ˜P 8 Pworld, homo ← concatenate([Pworld, 1Hp×Wp ]) 9 P′ world, homo ← T · Pworld, homo 10 P′ world ← P′ world, homo[:, :, : 3] 11 T12 ← K · P′ world 12 return T12 Point trajectory extraction from videos. We use CoTracker Karaev et al. (2023) to extract p...

    2023

  26. [37]

    Table 4: Ablations on training datasets

    17 Figure 12: Point trajectory estimation results. Table 4: Ablations on training datasets. Dataset Setting Training steps ATE (m, ↓) RPE trans (m, ↓) RPE Rot (deg, ↓) FID (↓) 10k real-world 40k 0.0396 0.0232 0.1939 103.5 10k games 40k 0.0421 0.0211 0.2139 105.3 10k real-world...

    2024

  27. [38]

    For all tasks, the inputs to the network are the first frame and the extracted trajectories

    First frame Trajectories Warp trajectories by camera view Edited first frame Trajectories (a) Input process visualization of camera motion control on videos (b) Input process visualization of video editing Figure 13: Input process visualization. For all tasks, the inputs to th...

    2024

  28. [39]

    16, illustrating results from five scenes with a single camera trajectory

    A.9 C OMPARISON ON CAMERA TRAJECTORIES For clearer understanding, we visualize the predicted trajectories in Fig. 16, illustrating results from five scenes with a single camera trajectory. The figure shows that, thanks to the explicit modeling of camera motion, our method’s es...

    2024

  29. [2022]

    Cogvideo: Large-scale pre- training for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868,

    arXiv

  30. [2023]

    Videocrafter1: Open diffusion models for high- quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.19512, 2023a. Haoxin Chen, Yong Zhang, Xi...

    arXiv

  31. [2024]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.