Pith. sign in

REVIEW 4 major objections 6 minor 103 references

FlexComposer turns video compositing into a single trajectory-guided generation step, handling both static images and pre-animated footage without explicit 3D reconstruction or auxiliary control adapters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:11 UTC pith:4TLNYP4T

load-bearing objection Clever, honest engineering paper on trajectory-guided compositing, held back by an unmeasured latent-equivariance assumption; deserves review but needs a direct equivariance check. the 4 major comments →

arxiv 2607.29627 v1 pith:4TLNYP4T submitted 2026-07-31 cs.CV

FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control

Songchun Zhang , Sitong Guo , Xianghao Kong , Pengwei Liu , Yuwei Guo , Lvmin Zhang , Anyi Rao This is my paper
classification cs.CV
keywords video compositingtrajectory controllatent injectiontranslation equivariancevideo diffusioncanonical foreground representationsynthetic-to-real curriculumvideo harmonization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to prove that video compositing—placing an external asset into a background video—can be reframed as one trajectory-guided conditional generation task that works equally for static images and already-animated video clips. Its claim rests on three designs: a canonical foreground representation that separates the object's own motion from its global displacement, a parameter-free latent-injection mechanism that moves the object by shifting its VAE features along the user's path, and a synthetic-to-real curriculum that teaches the generative model to harmonize lighting and shadows implicitly. If these claims hold, multi-stage VFX pipelines for insertion can collapse into a single generative pass, and dynamic assets keep their intrinsic motion while following new paths—something image-animation methods cannot do. A sympathetic reader would care because this would make controlled compositing both simpler and more faithful, for product photos and for footage alike.

Core claim

The central discovery is that a video VAE's latent space can be treated as translation-equivariant to the precision needed for direct feature transport: canonical foreground features are copied from their centered position to target trajectory coordinates via Eq. (2), with the temporal index coupled on both sides so intrinsic motion survives the move. Because the injection is a pure coordinate shift, no learnable control adapter is needed, and the paper argues this avoids the signal degradation that adapters introduce. The Unified Canonical Foreground Representation—centering dynamic videos, expanding static images with temporal noise—makes diverse inputs uniform for this same mechanism, and

What carries the argument

The load-bearing mechanism is Spatial-Aware Latent Injection, expressed in Eq. (2): after the foreground is stabilized into a canonical latent sequence, the user's 3D trajectory is projected to 2D latent coordinates (Eq. 1) and the object's latent features are transported to each trajectory position by a direct coordinate shift, gated by a visibility mask that handles occlusions. The justification for this parameter-free transport is the translation equivariance of the video VAE latent space, which lets a shift replace any learnable adapter. Two supporting mechanisms carry the rest of the argument: the Unified Canonical Foreground Representation (motion decoupling and static expansion with G

Load-bearing premise

The whole mechanism rests on the assumption that moving a foreground's latent features by an integer offset in the latent grid places the semantically correct content at the target location, for every frame and every offset—a near-exact translation equivariance of the video VAE that the paper does not directly measure.

What would settle it

Encode a sharp object at two positions differing by one latent-grid unit; if the difference between the shifted encoding and the encoding of the shifted image is not near zero, or if decoding the transported latent blurs or smears object boundaries, the parameter-free transport fails. Additionally, running a trajectory with sub-latent-unit per-frame motion should produce a stationary or wobbling object if the nearest-integer rounding in Eq. (1) erases the motion—tracking the centroid would reveal it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single static product photo can be animated along any user-defined path while preserving identity, with the model hallucinating plausible micro-motion and view changes.
  • Pre-animated footage—an explosion, a fluttering wing—can be repositioned onto a new trajectory without losing its internal dynamics, a capability image-animation baselines lack.
  • Occlusion is explicitly controllable through the visibility gate: the asset disappears behind occluders instead of ghosting through them.
  • Because motion is decoupled from appearance at the representation level, the same framework supports non-pixel-aligned motion transfer from a source subject to a different target subject in a new background.
  • Lighting, shadows, and reflections emerge from the curriculum rather than from a dedicated harmonization stage, improving perceptual consistency over color-only harmonization methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: If translation equivariance is as strong as the method assumes, the parameter-free injection logic should transfer to other latent video diffusion models with only backbone fine-tuning, suggesting a general recipe for trajectory-controlled editing beyond compositing.
  • Inference: The representation's lack of geometric warping implies a testable boundary: motion transfer quality should degrade as source and target subjects diverge in shape; a systematic shape-difference sweep would map this boundary.
  • Inference: The paper's stated limitation that physics are not simulated points to an immediate extension—replacing the visibility gate with simple depth ordering or a lightweight physics engine—which could remove ghosting and implausible interactions.
  • Inference: The reported training budget (roughly 8k steps over three curriculum stages) leaves open whether trajectory adherence scales smoothly with more real footage or denser trajectory supervision, an experiment worth running for new domains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents FlexComposer, a video compositing framework that inserts static images or dynamic clips into background videos along user-specified 3D trajectories. The method has three components: pixel-space canonicalization (stabilization for video, expansion/noise injection for images), a `parameter-free` latent injection that copies VAE features from a canonical location to trajectory positions using assumed translation equivariance, and a synthetic-to-real curriculum. The system is built on Wan 2.1 I2V with LoRA fine-tuning. Experiments compare against I2V trajectory-control baselines, V2V compositing/editing baselines, and harmonization methods, reporting EPE, FVD, PSNR, SSIM, and VBench metrics.

Significance. If the central mechanism is validated, the paper offers a useful unified treatment of two previously separate settings (static and dynamic foregrounds), with an appealingly simple control mechanism. The extensive ablations and user study are strengths. However, the key novelty rests on an unverified equivariance property, and the evaluation protocol has circularity and reference-frame ambiguities. With additional analysis and more rigorous evaluation, the contribution would be publishable; in current form the headline claims are not fully supported.

major comments (4)
  1. [§3.2, Eqs. (1)-(2)] The parameter-free transport operation is the paper's central novelty, but it assumes the Wan video VAE is translation-equivariant under integer latent shifts for all foreground content and across temporally compressed latent frames. Strided convolutions, patchification, and temporal compression make exact equivariance unlikely; Eq. (1) also rounds coordinates, introducing up to 0.5/f_s pixel error. The paper provides no measurement of equivariance, no comparison of C_fuse to the VAE encoding of a pixel-space ground-truth composite, and no sensitivity analysis for rounding or fast motion. Consequently the reported advantage over learned adapters may reflect LoRA-finetuned compensation rather than faithful transport. I ask for a direct equivariance test (e.g., encode shifted inputs and measure feature-shift error), a C_fuse-vs-E(pixel composite) comparison, and an analysis of how rounding
  2. [§A.4, §B.2, Tables 1-3] EPE is computed by tracking generated centroids with SpatialTrackerV2, and the same tracker is used to generate pseudo-ground-truth trajectories for Phase 2 and Phase 3 training data, as well as for the canonical stabilization in §3.1. This creates correlated evaluation: systematic tracker errors are shared between the training target and the evaluation measure, potentially inflating trajectory-adherence gains. Also, Tables 1-3 report point estimates only, with no confidence intervals or significance tests, and the V2V benchmark is a private 50-clip set with no release plan. Please evaluate with an independent tracker or human-annotated trajectories, release the benchmark, and report variance.
  3. [§4.2, Tables 1-2] PSNR and SSIM are reported for trajectory-controlled generation, but the reference video for these metrics is not defined. In the static foreground compositing setting there is no ground-truth composite; if the reference is the unmodified background video, inserting any foreground will lower PSNR/SSIM, making the numbers uninterpretable; if a composited ground truth is used, its construction must be described. Please specify the reference and, ideally, replace these metrics with task-appropriate perceptual tests.
  4. [Table 4, §C.2] The ablation labeled 'w/o Latent Transport' is described in the supplementary as replacing the transport module with a learned convolutional encoder, not as removing transport entirely. The main text's discussion ('sliding artifacts,' 'fail to anchor') reads as if the module were absent. As presented, the ablation cannot isolate whether the proposed transport is faithful or whether the LoRA-finetuned backbone learns corrections. Please align the label and description and add an ablation that varies only the transport operation while holding the condition signal and training protocol constant.
minor comments (6)
  1. [Abstract/§3.2, §A.1] 'Without auxiliary learnable adapters' is potentially misleading because LoRA adapters (rank 64) are trained on the backbone. Clarify that 'parameter-free' refers only to the injection function, not the overall system.
  2. [Eq. (1)] Define f_s, f_t and the nearest-integer symbol before using; also justify the temporal averaging over f_t frames, which may alias fast global motion.
  3. [Table 4] The label 'w/o Noise in Static Exp.' is later called 'w/o Static Noise' in the text; use consistent terminology.
  4. [§4.3] Provide details of the 50-clip V2V test set (sources, trajectory distributions, masking protocol) or release it.
  5. [§D] The user study of 25 evaluators × 20 cases should report confidence intervals; 'dominant lead' overstates precision.
  6. [References] The text cites Grounded-SAM2 as [35], but [35] is Segment Anything; check the citation mapping. Similarly, ensure SpatialTracker v2 citations are consistent.

Circularity Check

0 steps flagged

No significant circularity: the core latent-transport mechanism is an explicit coordinate mapping, and the reported evaluations are external video-quality and trajectory metrics.

full rationale

The central derivation, Eq. (2), is a direct coordinate-shift operation in VAE latent space: for each offset delta, canonical features are copied from c+delta to u_n+delta. This is an explicit constructive definition of the injected condition, not a fitted parameter renamed as a prediction. The trajectory appears in the conditioning signal by construction, but the paper's claim is about the generated video following that condition, which is evaluated on held-out video quality and trajectory metrics; that is a learned empirical outcome rather than an equivalence to the input. The training objective (Eq. 4) is a standard flow-matching loss against ground-truth video velocity, so no fitted scalar is being re-presented as a first-principles result. The initialization from Wan-Move weights and the use of LoRA are engineering choices and are compared against Wan-Move as an external baseline, so they do not reduce the central claim to a self-citation. Two caveats are worth stating but are not circularity: (1) the paper asserts, without measuring, that the Wan VAE latent space is translation-equivariant to the precision required by Eq. (2); if false, the method underperforms, but this is an unverified empirical assumption, not an input-output identity. (2) SpatialTrackerV2 is used both to generate pseudo-ground-truth trajectories for the generative training data (App. B.3) and to compute EPE by tracking the output centroid (App. A.4); this makes EPE somewhat tracker-internal, but the test-time reference is the user/benchmark trajectory, so the comparison is not the tracker predicting its own fitted output. These are robustness/evaluation concerns, not circular derivation steps.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central contribution is not a closed-form derivation but a trained system, so the ledger is dominated by assumptions about latent-space geometry and training transfer. The most fragile entries are the unmeasured VAE translation equivariance and the centroid-translation decomposition, both of which the parameter-free injection pipeline leans on directly. The hyperparameters are hand-chosen but not deeply coupled to the core claim. No physical entities are introduced; the 'canonical representation' and 'latent transport' are internal machinery whose independent evidence is weak.

free parameters (4)
  • LoRA rank / alpha = r=64, alpha=32
    Chosen for the adaptation capacity of the frozen Wan DiT; no sensitivity study is reported, and the central composition capability depends on this hyperparameter setting.
  • Motion dropout probability p = 0.05
    Hand-set to balance preserving the base I2V prior against trajectory adherence; no ablation on this probability is given.
  • Dense-to-sparse trajectory annealing endpoints = 200 -> 20 trajectory points
    The schedule Kmax(t) is manually chosen; the final sparsity level affects how robust the model is to sparse user trajectories at inference.
  • Curriculum step counts and dataset sizes = 2k/5k/1k steps; 12k/54k/5k clips
    Stage durations and data volumes are hand-chosen; no experiments show that these ratios are optimal, and the central generalization claim depends on them.
axioms (4)
  • domain assumption VAE latent translation equivariance
    Eq. (2) moves canonical features by coordinate offsets in the latent space and assumes the result places the foreground correctly. Wan's VAE uses strided convolutions and temporal compression; exact translation equivariance is not established or measured in the paper.
  • domain assumption Centroid-stabilization fully decouples global displacement from intrinsic motion
    Section 3.1 recenters the object by tracking its centroid and applying a reverse translation. This assumes global motion is purely translational; rotations, scaling, and viewpoint changes are not removed by this step.
  • domain assumption Temporal averaging in Eq. (1) faithfully represents fast trajectories
    Latent frame coordinates are computed as the average of projected positions over ft input frames. High-frequency motion within one latent frame is collapsed to an average, which can alias fast or oscillating trajectories.
  • domain assumption LoRA fine-tuning preserves the base model's generative priors while learning composition
    The paper freezes Wan's DiT and trains only low-rank adapters (App. A.1). Ablations support the design, but there is no formal or empirical guarantee that the adapted model does not rely on corrupted or overwritten priors.
invented entities (2)
  • Unified Canonical Foreground Representation (Z_can) no independent evidence
    purpose: A stabilized, centered latent sequence that standardizes static images, dynamic clips, and diverse object types before transport onto user trajectories.
    This is an internal representational construct with no external falsifiable handle; its effectiveness is only evidenced by the paper's own ablations and metrics.
  • Spatial-Aware Latent Injection (transport operation) no independent evidence
    purpose: Moves canonical latent features to trajectory coordinates using claimed VAE translation equivariance, replacing auxiliary adapters such as ControlNet.
    The mechanism is validated only within the paper's experiments; no independent implementation, theoretical bound, or release demonstrates the equivariance it depends on.

pith-pipeline@v1.3.0-daily-deepseek · 19209 in / 13796 out tokens · 169503 ms · 2026-08-03T03:11:21.249069+00:00 · methodology

0 comments
read the original abstract

Generative video compositing, which involves inserting external assets seamlessly into existing video sequences, is essential for content creation and visual effects. However, existing approaches suffer from a control-fidelity trade-off: they either hallucinate motion from static images, failing to preserve the dynamics of pre-animated assets, or lack fine-grained spatial control for precise asset placement along user-defined trajectories. We propose FlexComposer, a unified framework that standardizes video compositing as a trajectory-guided conditional generation task, enabling the seamless integration of both static images and dynamic footage. Our approach introduces three key designs: (1) a Unified Canonical Foreground Representation that decouples an object's intrinsic motion from its global displacement, standardizing heterogeneous inputs into a stabilized, centered latent space; (2) a Spatial-Aware Latent Injection strategy that exploits the translation equivariance of VAE latent spaces to transport canonical features onto target trajectories via a parameter-free mechanism; and (3) a Hybrid Dataset and Synthetic-to-Real Curriculum that synergizes procedural simulation, real-world cinematic footage, and generative data to implicitly learn physically plausible illumination and shadow harmonization. This unified design handles diverse inputs from product photos to dynamic subjects achieving high-fidelity motion control and environmental integration without the need for explicit 3D reconstruction or auxiliary learnable adapters. Extensive experiments demonstrate that FlexComposer outperforms state-of-the-art methods in visual quality, temporal consistency, and trajectory adherence.

Figures

Figures reproduced from arXiv: 2607.29627 by Anyi Rao, Lvmin Zhang, Pengwei Liu, Sitong Guo, Songchun Zhang, Xianghao Kong, Yuwei Guo.

Figure 1
Figure 1. Figure 1: Versatile Video Compositing with FlexComposer. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Pipeline of FlexComposer. (1) Inputs: A dynamic background, a static/dynamic foreground asset, a 3D trajectory, and visibility masks. (2) Unified Representation: Diverse inputs are encoded into a canonical feature space to preserve identity and local motion. (3) Spatiotemporal Latent Injection: Canonical features are warped into the background latent space along the trajectory, utilizing a visibility g… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Comparison on Static Foreground Compositing. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Additional Qualitative Comparison on Static Foreground Compositing. We compare our method against trajectory-controlled I2V methods on two challenging scenarios [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative Comparison on Dynamic Foreground Compositing. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Harmonization Results. Our method synthesizes shadows and reflections alongside color adaptation [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visual Ablation Analysis. Im￾pact of noise injection in static image ex￾pansion. W/o Vis. Gate Ours W/o BG Context [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative results of complex 3D rotations. FlexComposer synthesizes chal￾lenging out-of-plane motions (e.g., flips, rolls) by applying explicit 3D transformations to a SAM3D-reconstructed proxy. Injecting the resulting trajectory ensures temporally consistent synthesis even during extreme pose changes. Canonical Representation & Noise. Removing the canonical pipeline (w/o Canonical Repr.) causes severe m… view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative results of motion transfer. By mapping source trajectories to tar￾get characters via latent transport, our framework facilitates motion synthesis in new background. This approach supports non-pixel-aligned transfer, allowing for motion transfer across different character and backgrounds. the perceptual quality significantly degrades as objects appear inconsistent with the target scene lighting… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

103 extracted references · 36 linked inside Pith

  1. [1]

    Adobe: After effects.https://www.adobe.com/products/aftereffects.html (2025), accessed: 2025-01-03 4

    2025

  2. [2]

    Adobe Systems Inc.: Mixamo.https://www.mixamo.com(2025) 7, 2

    2025

  3. [3]

    arXiv preprint arXiv:2509.19296 (2025) 4

    Bahmani, S., Shen, T., Ren, J., Huang, J., Jiang, Y., Turki, H., Tagliasacchi, A., Lindell, D.B., Gojcic, Z., Fidler, S., et al.: Lyra: Generative 3d scene reconstruction via video diffusion model self-distillation. arXiv preprint arXiv:2509.19296 (2025) 4

    arXiv 2025

  4. [4]

    arXiv preprint arXiv:2503.11647 (2025) 4

    Bai, J., Xia, M., Fu, X., Wang, X., Mu, L., Cao, J., Liu, Z., Hu, H., Bai, X., Wan, P., et al.: Recammaster: Camera-controlled generative rendering from a single video. arXiv preprint arXiv:2503.11647 (2025) 4

    Pith/arXiv arXiv 2025

  5. [5]

    arXiv preprint arXiv:2412.07760 (2024) 4

    Bai, J., Xia, M., Wang, X., Yuan, Z., Fu, X., Liu, Z., Hu, H., Wan, P., Zhang, D.: Syncammaster: Synchronizing multi-camera video generation from diverse view- points. arXiv preprint arXiv:2412.07760 (2024) 4

    Pith/arXiv arXiv 2024

  6. [6]

    arXiv preprint arXiv:2606.04527 (2026) 3

    Bian, Y., Xue, Z., Zhang, S., Zhang, S., Jin, W., Li, Y., Zhuang, J., Li, H., Huang, J., Huang, H., et al.: Echo-infinity: Learning evolving memory for real-time infinite video generation. arXiv preprint arXiv:2606.04527 (2026) 3

    Pith/arXiv arXiv 2026

  7. [7]

    arXiv preprint arXiv:2311.15127 (2023) 3

    Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023) 3

    Pith/arXiv arXiv 2023

  8. [8]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18392–18402 (2023) 4 16 Songchun Zhang et al

    2023

  9. [9]

    OpenAI Technical Report (2024),https://openai.com/research/ video-generation-models-as-world-simulators3

    Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., et al.: Video generation models as world simulators. OpenAI Technical Report (2024),https://openai.com/research/ video-generation-models-as-world-simulators3

    2024

  10. [10]

    arXiv preprint arXiv:2506.23361 (2025) 2, 4

    Cai,Y.,Zhang,H.,Chen,X.,Xing,J.,Hu,Y.,Zhou,Y.,Zhang,K.,Zhang,Z.,Kim, S.Y., Wang, T., et al.: Omnivcus: Feedforward subject-driven video customization with multimodal control conditions. arXiv preprint arXiv:2506.23361 (2025) 2, 4

    arXiv 2025

  11. [11]

    arXiv preprint arXiv:2601.02358 (2026) 2, 4

    Chen, J., He, T., Fu, Z., Wan, P., Gai, K., Ye, W.: Vino: A unified visual generator with interleaved omnimodal context. arXiv preprint arXiv:2601.02358 (2026) 2, 4

    arXiv 2026

  12. [12]

    arXiv preprint arXiv:2311.00213 (2023) 4

    Cheng, J., Xiao, T., He, T.: Consistent video-to-video transfer using synthetic dataset. arXiv preprint arXiv:2311.00213 (2023) 4

    Pith/arXiv arXiv 2023

  13. [13]

    arXiv preprint arXiv:2512.08765 (2025) 4, 8, 9, 10

    Chu, R., He, Y., Chen, Z., Zhang, S., Xu, X., Xia, B., Wang, D., Yi, H., Liu, X., Zhao, H., et al.: Wan-move: Motion-controllable video generation via latent trajectory guidance. arXiv preprint arXiv:2512.08765 (2025) 4, 8, 9, 10

    arXiv 2025

  14. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873–12883 (2021) 3

    2021

  15. [15]

    Foundry: Nuke.https://www.foundry.com/products/nuke(2025), accessed: 2025-01-03 4

    2025

  16. [16]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Gao, C., Saraf, A., Kopf, J., Huang, J.B.: Dynamic view synthesis from dynamic monocular video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5712–5721 (2021) 4

    2021

  17. [17]

    arXiv preprint arXiv:2412.02700 (2024) 4, 6

    Geng, D., Herrmann, C., Hur, J., Cole, F., Zhang, S., Pfaff, T., Lopez-Guevara, T., Doersch, C., Aytar, Y., Rubinstein, M., et al.: Motion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700 (2024) 4, 6

    Pith/arXiv arXiv 2024

  18. [18]

    Advances in neural in- formation processing systems27(2014) 3

    Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural in- formation processing systems27(2014) 3

    2014

  19. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D.J., Gnanapragasam, D., Golemo, F., Herrmann, C., et al.: Kubric: A scalable dataset generator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3749–3761 (2022) 7, 2

    2022

  20. [20]

    In: Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers

    Gu, Z., Yan, R., Lu, J., Li, P., Dou, Z., Si, C., Dong, Z., Liu, Q., Lin, C., Liu, Z., et al.: Diffusion as shader: 3d-aware video diffusion for versatile video generation control. In: Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers. pp. 1–12 (2025) 4, 6

    2025

  21. [21]

    In: ECCV

    Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., Dai, B.: Sparsectrl: Adding sparse controls to text-to-video diffusion models. In: ECCV. pp. 330–348 (2024) 4

    2024

  22. [22]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Guo, Z., Han, X., Zhang, J., Shan, S., Zheng, H.: Video harmonization with triplet spatio-temporal variation patterns. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19177–19186 (2024) 2, 4, 11

    2024

  23. [23]

    He, H., Xu, Y., Guo, Y., Wetzstein, G., Dai, B., Li, H., Yang, C.: Cameractrl: En- ablingcameracontrolfortext-to-videogeneration.arXivpreprintarXiv:2404.02101 (2024) 4

    Pith/arXiv arXiv 2024

  24. [24]

    Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference- free evaluation metric for image captioning (2022),https://arxiv.org/abs/2104. 0871811

    2022

  25. [25]

    Advances in neural information processing systems30(2017) 9 Flexcomposer 17

    Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30(2017) 9 Flexcomposer 17

    2017

  26. [26]

    NeurIPS (2020) 3

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurIPS (2020) 3

    2020

  27. [27]

    Advances in neural information processing systems35, 8633– 8646 (2022) 3

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. Advances in neural information processing systems35, 8633– 8646 (2022) 3

    2022

  28. [28]

    ICLR1(2), 3 (2022) 8, 1

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022) 8, 1

    2022

  29. [29]

    arXiv preprint arXiv:2312.02919 (2023) 4

    Huang, H.P., Su, Y.C., Sun, D., Jiang, L., Jia, X., Zhu, Y., Yang, M.H.: Fine- grained controllable video generation via object appearance and context. arXiv preprint arXiv:2312.02919 (2023) 4

    Pith/arXiv arXiv 2023

  30. [30]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al.: Vbench: Comprehensive benchmark suite for video gener- ative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21807–21818 (2024) 10

    2024

  31. [31]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Jiang, Z., Han, Z., Mao, C., Zhang, J., Pan, Y., Liu, Y.: Vace: All-in-one video creation and editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 17191–17202 (2025) 2, 4, 10, 11

    2025

  32. [32]

    arXiv preprint arXiv:2509.20360 (2025) 2, 4

    Ju, X., Wang, T., Zhou, Y., Zhang, H., Liu, Q., Zhao, N., Zhang, Z., Li, Y., Cai, Y., Liu, S., et al.: Editverse: Unifying image and video editing and generation with in-context learning. arXiv preprint arXiv:2509.20360 (2025) 2, 4

    Pith/arXiv arXiv 2025

  33. [33]

    ACM Transac- tions on Graphics (TOG)33(3), 1–15 (2014) 2, 4

    Karsch, K., Sunkavalli, K., Hadap, S., Carr, N., Jin, H., Fonte, R., Sittig, M., Forsyth, D.: Automatic scene inference for 3d object compositing. ACM Transac- tions on Graphics (TOG)33(3), 1–15 (2014) 2, 4

    2014

  34. [34]

    In: European conference on computer vision

    Ke, Z., Sun, C., Zhu, L., Xu, K., Lau, R.W.: Harmonizer: Learning to perform white-box image and video harmonization. In: European conference on computer vision. pp. 690–706. Springer (2022) 2, 11

    2022

  35. [35]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023) 5

    2023

  36. [36]

    arXiv preprint arXiv:2412.03603 (2024) 3

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., et al.: Hunyuanvideo: A system- atic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024) 3

    Pith/arXiv arXiv 2024

  37. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kong, X., Zhang, Z., Guo, Y., Zhao, Z., Zhang, S., Rao, A.: Composing con- cepts from images and videos via concept-prompt binding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14800– 14810 (2026) 2, 4

    2026

  38. [38]

    arXiv preprint arXiv:2403.14468 (2024) 4, 10, 11

    Ku, M., Wei, C., Ren, W., Yang, H., Chen, W.: Anyv2v: A tuning-free framework for any video-to-video editing tasks. arXiv preprint arXiv:2403.14468 (2024) 4, 10, 11

    Pith/arXiv arXiv 2024

  39. [39]

    Kuaishou: Keling (2024),https://kling.kuaishou.com/3

    2024

  40. [40]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Lee, Y.C., Lu, E., Rumbley, S., Geyer, M., Huang, J.B., Dekel, T., Cole, F.: Gener- ative omnimatte: Learning to decompose video into layers. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 12522–12532 (2025) 4

    2025

  41. [41]

    In: European Conference on Computer Vision

    Lee, Y.C., Zhang, Z., Blackburn-Matzen, K., Niklaus, S., Zhang, J., Huang, J.B., Liu, F.: Fast view synthesis of casual videos with soup-of-planes. In: European Conference on Computer Vision. pp. 278–296. Springer (2024) 4

    2024

  42. [42]

    arXiv preprint arXiv:2411.10836 (2024) 4 18 Songchun Zhang et al

    Lei, G., Wang, C., Li, H., Zhang, R., Wang, Y., Xu, W.: Animateanything: Consistent and controllable animation for video generation. arXiv preprint arXiv:2411.10836 (2024) 4 18 Songchun Zhang et al

    Pith/arXiv arXiv 2024

  43. [43]

    arXiv preprint arXiv:2503.16421 (2025) 9, 10

    Li, Q., Xing, Z., Wang, R., Zhang, H., Dai, Q., Wu, Z.: Magicmotion: Control- lable video generation with dense-to-sparse trajectory guidance. arXiv preprint arXiv:2503.16421 (2025) 9, 10

    arXiv 2025

  44. [44]

    arXiv preprint arXiv:2406.15339 (2024) 9, 10

    Li, Y., Wang, X., Zhang, Z., Wang, Z., Yuan, Z., Xie, L., Zou, Y., Shan, Y.: Image conductor: Precision control for interactive video synthesis. arXiv preprint arXiv:2406.15339 (2024) 9, 10

    Pith/arXiv arXiv 2024

  45. [45]

    arXiv preprint arXiv:2210.02747 (2022) 7

    Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022) 7

    Pith/arXiv arXiv 2022

  46. [46]

    arXiv preprint arXiv:2511.01678 (2025) 6

    Liu, R., Yuan, H., Dong, B., Xing, J., Wang, J., Zhao, R., Xing, Y., Chen, W., Wang, F.: Unilumos: Fast and unified image and video relighting with physics- plausible feedback. arXiv preprint arXiv:2511.01678 (2025) 6

    arXiv 2025

  47. [47]

    arXiv preprint arXiv:2205.00687 (2022) 11

    Lu, X., Huang, S., Niu, L., Cong, W., Zhang, L.: Deep video harmonization with color mapping consistency. arXiv preprint arXiv:2205.00687 (2022) 11

    Pith/arXiv arXiv 2022

  48. [48]

    In: SIGGRAPH Asia (2024) 4

    Ma,W.D.K.,Lewis,J.P.,Kleijn,W.B.:Trailblazer:Trajectorycontrolfordiffusion- based video generation. In: SIGGRAPH Asia (2024) 4

    2024

  49. [49]

    arXiv preprint arXiv:2506.04590 (2025) 4

    Ma, Y., Feng, K., Zhang, X., Liu, H., Zhang, D.J., Xing, J., Zhang, Y., Yang, A., Wang, Z., Chen, Q.: Follow-your-creation: Empowering 4d creation through video inpainting. arXiv preprint arXiv:2506.04590 (2025) 4

    Pith/arXiv arXiv 2025

  50. [50]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Ma, Y., He, Y., Cun, X., Wang, X., Chen, S., Li, X., Chen, Q.: Follow your pose: Pose-guided text-to-video generation using pose-free videos. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 4117–4125 (2024) 4

    2024

  51. [51]

    arXiv preprint arXiv:2512.16920 (2025) 4

    Mai, J., Wang, C., Qian, G.G., Menapace, W., Tulyakov, S., Ghanem, B., Wonka, P.,Mirzaei,A.:Easyv2v:Ahigh-qualityinstruction-basedvideoeditingframework. arXiv preprint arXiv:2512.16920 (2025) 4

    arXiv 2025

  52. [52]

    arXiv preprint arXiv:2405.13865 (2024) 4

    Mou, C., Cao, M., Wang, X., Zhang, Z., Shan, Y., Zhang, J.: Revideo: Remake a video with motion and content control. arXiv preprint arXiv:2405.13865 (2024) 4

    Pith/arXiv arXiv 2024

  53. [53]

    arXiv preprint arXiv:2411.04989 (2024) 4

    Namekata, K., Bahmani, S., Wu, Z., Kant, Y., Gilitschenski, I., Lindell, D.B.: Sg-i2v: Self-guided trajectory control in image-to-video generation. arXiv preprint arXiv:2411.04989 (2024) 4

    Pith/arXiv arXiv 2024

  54. [54]

    Pandey, K., Gadelha, M., Hold-Geoffroy, Y., Singh, K., Mitra, N.J., Guerrero, P.: Motion modes: What could happen next? arXiv preprint arXiv:2412.00148 (2024) 4

    Pith/arXiv arXiv 2024

  55. [55]

    Peebles, W., Xie, S.: Scalable diffusion models with transformers (2023),https: //arxiv.org/abs/2212.097483

    Pith/arXiv arXiv 2023

  56. [56]

    ACM SIGGRAPH Computer Graphics18(3), 253–259 (1984) 4

    Porter, T., Duff, T.: Compositing digital images. ACM SIGGRAPH Computer Graphics18(3), 253–259 (1984) 4

    1984

  57. [57]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15932–15942 (2023) 4

    2023

  58. [58]

    ArXivabs/2401.14159(2024),https://api.semanticscholar.org/CorpusID: 2672120474

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks. ArXivabs/2401.14159(2024),https://api.semanticscholar.org/CorpusID: 2672120474

    Pith/arXiv arXiv 2024

  59. [59]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Ren, X., Shen, T., Huang, J., Ling, H., Lu, Y., Nimier-David, M., Müller, T., Keller, A., Fidler, S., Gao, J.: Gen3c: 3d-informed world-consistent video gener- ation with precise camera control. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 6121–6132 (2025) 4

    2025

  60. [60]

    In: CVPR (2022) 3 Flexcomposer 19

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022) 3 Flexcomposer 19

    2022

  61. [61]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Sun, W., Chen, S., Liu, F., Chen, Z., Duan, Y., Zhu, J., Zhang, J., Wang, Y.: Dimensionx: Create any 3d and 4d scenes from a single image with decoupled video diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13695–13706 (2025) 4

    2025

  62. [62]

    arXiv preprint arXiv:2512.16776 (2025) 2, 4

    Team, K., Chen, J., Ci, Y., Du, X., Feng, Z., Gai, K., Guo, S., Han, F., He, J., He, K., et al.: Kling-omni technical report. arXiv preprint arXiv:2512.16776 (2025) 2, 4

    Pith/arXiv arXiv 2025

  63. [63]

    In: Proceedings of the Spe- cial Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers

    Tu, Y., Luo, H., Chen, X., Ji, S., Bai, X., Zhao, H.: Videoanydoor: High-fidelity video object insertion with precise motion control. In: Proceedings of the Spe- cial Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers. pp. 1–11 (2025) 2, 4

    2025

  64. [64]

    arXiv preprint arXiv:1812.01717 (2018) 9

    Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 (2018) 9

    Pith/arXiv arXiv 2018

  65. [65]

    arXiv preprint arXiv:2503.20314 (2025) 6, 8, 1

    Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025) 6, 8, 1

    Pith/arXiv arXiv 2025

  66. [66]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wang, C., Zhuang, P., Ngo, T.D., Menapace, W., Siarohin, A., Vasilkovsky, M., Skorokhodov, I., Tulyakov, S., Wonka, P., Lee, H.Y.: 4real-video: Learning gener- alizable photo-realistic 4d video diffusion. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 17723–17732 (2025) 4

    2025

  67. [67]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wang, H., Ouyang, H., Wang, Q., Wang, W., Cheng, K.L., Chen, Q., Shen, Y., Wang, L.: Levitor: 3d trajectory oriented image-to-video synthesis. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 12490–12500 (2025) 9, 10

    2025

  68. [68]

    arXiv preprint arXiv:2512.16924 (2025) 4

    Wang, H., Ouyang, H., Wang, Q., Yu, Y., Meng, Y., Wang, W., Cheng, K.L., Ma, S., Bai, Q., Li, Y., et al.: The world is your canvas: Painting promptable events with reference images, trajectories, and text. arXiv preprint arXiv:2512.16924 (2025) 4

    arXiv 2025

  69. [69]

    arXiv preprint arXiv:2402.01566 (2024) 4

    Wang, J., Zhang, Y., Zou, J., Zeng, Y., Wei, G., Yuan, L., Li, H.: Boxima- tor: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566 (2024) 4

    Pith/arXiv arXiv 2024

  70. [70]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, Q., Ye, V., Gao, H., Zeng, W., Austin, J., Li, Z., Kanazawa, A.: Shape of motion: 4d reconstruction from a single video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9660–9672 (2025) 4

    2025

  71. [71]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, W., Song, H., Zhao, S., Shen, J., Zhao, S., Hoi, S.C., Ling, H.: Learning unsupervised video object segmentation through visual attention. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3064–3074 (2019) 9

    2019

  72. [72]

    Advances in Neural Information Processing Systems36(2024) 4

    Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D., Zhou, J.: Videocomposer: Compositional video synthesis with motion control- lability. Advances in Neural Information Processing Systems36(2024) 4

    2024

  73. [73]

    IEEE transactions on image processing 13(4), 600–612 (2004) 9, 11

    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004) 9, 11

    2004

  74. [74]

    In: SIGGRAPH (2024) 4

    Wang,Z.,Yuan,Z.,Wang,X.,Li,Y.,Chen,T.,Xia,M.,Luo,P.,Shan,Y.:Motionc- trl: A unified and flexible motion controller for video generation. In: SIGGRAPH (2024) 4

    2024

  75. [75]

    arXiv preprint arXiv:2510.08377 (2025) 2 20 Songchun Zhang et al

    Wei, C., Liu, Q., Ye, Z., Wang, Q., Wang, X., Wan, P., Gai, K., Chen, W.: Uni- video: Unified understanding, generation, and editing for videos. arXiv preprint arXiv:2510.08377 (2025) 2 20 Songchun Zhang et al

    Pith/arXiv arXiv 2025

  76. [76]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-a-video: One-shot tuning of image diffusion models for text- to-video generation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 7623–7633 (2023) 4

    2023

  77. [77]

    In: Proceed- ings of the Computer Vision and Pattern Recognition Conference

    Wu, R., Gao, R., Poole, B., Trevithick, A., Zheng, C., Barron, J.T., Holynski, A.: Cat4d: Create anything in 4d with multi-view video diffusion models. In: Proceed- ings of the Computer Vision and Pattern Recognition Conference. pp. 26057–26068 (2025) 4

    2025

  78. [78]

    In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision

    Wu, Y., Chen, L., Li, R., Wang, S., Xie, C., Zhang, L.: Insvie-1m: Effective instruction-based video editing with elaborate dataset construction. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision. pp. 16692– 16701 (2025) 4

    2025

  79. [79]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Xian, W., Huang, J.B., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9421–9431 (2021) 4

    2021

  80. [80]

    ArXiv abs/2507.12462(2025) 5

    Xiao, Y., Wang, J., Xue, N., Karaev, N., Makarov, Y., Kang, B., Zhu, X., Bao, H., Shen, Y., Zhou, X.: Spatialtrackerv2: 3d point tracking made easy. ArXiv abs/2507.12462(2025) 5

    Pith/arXiv arXiv 2025

Showing first 80 references.