Pith. sign in

REVIEW 3 major objections 6 minor 10 cited by

SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SynCamMaster claims that a frozen text-to-video diffusion model, augmented with per-block cross-view attention and camera-pose embeddings, can generate synchronized open-world videos from arbitrary viewpoints, and can also re-render a…

desk verdict A clearly-engineered step toward text-driven multi-camera video generation, but the evidence for the 'excellent inter-view synchronization' claim is thin: the cross-view module is frame-wise and the training data contains almost no true synchronized dynamic supervision. read the letter →

arxiv 2412.07760 v1 pith:K4LGUHLM submitted 2024-12-10 cs.CV

Jianhong Bai , Menghan Xia , Xintao Wang , Ziyang Yuan , Xiao Fu , Zuozhu Liu , Haoji Hu , Pengfei Wan
show 1 more author
Di Zhang
This is my paper · ORCID
classification cs.CV
keywords multi-cameravideogenerationtext-to-videodiffusioncross-viewattentioncameraposecontrol6-DoFviewpointsmulti-viewsynchronizationnovelviewsynthesishybridtrainingdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SynCamMaster is trying to establish that synchronized multi-camera video generation for open-world scenes can be obtained without per-scene optimization, by adding a small trainable module to a frozen text-to-video model. The authors' claim is that one text prompt plus a set of relative 6-DoF camera poses yields several videos of the same scene whose content and motion are consistent across viewpoints. This matters for virtual filming, because it replaces multi-camera setup and per-scene reconstruction with a single generative pass. The method's load-bearing mechanism is a cross-view attention module inserted into every transformer block, guided by camera-pose embeddings and trained on a deliberately mixed dataset of rendered, real, and regular videos.

What carries the argument

The multi-view synchronization module: at each DiT block, a 12-dimensional camera extrinsic vector [R,t] is embedded by a camera encoder and added to the spatial features of that view; then a cross-view self-attention layer aggregates features from the same frame across all n views; a linear projector and residual connection map the result back, so the frozen base model sees view-synchronized features. A progressive training schedule first exposes the model to small inter-view angles and gradually increases them, and the hybrid data mix (rendered synchronized videos for exact geometry, real multi-view images for open-domain diversity, monocular videos for quality) supplies the supervision that scarce multi-camera footage alone cannot.

What would settle it

Generate one scene from two cameras with a 120-degree azimuth separation and compare the estimated relative pose via feature matching with the input camera parameters while also counting correctly matched pixels; if matched-pixel counts fall to chance while pose error exceeds that of a single-view baseline, the synchronization module is not encoding true geometry.

Watch

Extended reading notes

Core claim

The central claim is that a frozen pre-trained text-to-video diffusion model can be turned into a multi-camera generator by inserting a lightweight multi-view synchronization module at every transformer block: a camera encoder maps normalized 6-DoF extrinsic parameters into per-view embeddings, a cross-view attention layer exchanges spatial features across all views at each frame, and residual projections return the aggregated features to the frozen network. Trained with a progressive schedule that starts with small viewpoint differences and widens them, plus a hybrid mix of rendered synchronized videos, real multi-view images, and monocular videos, the model generates n videos from a single text prompt that stay consistent in appearance and motion across arbitrary viewpoints. The paper also claims a direct extension: by replacing the first view's noisy latents with a clean reference video during inference, the same module re-renders that video from novel viewpoints.

Load-bearing premise

The frozen text-to-video model already contains enough learned 3D consistency and motion priors that cross-view attention on top of it can generalize to arbitrary open-world viewpoints.

Editorial extensions

If this is right

  • A single text prompt and a set of relative camera poses can produce n synchronized shots, eliminating per-scene optimization for multi-camera virtual filming.
  • Because the base model is frozen and only the new modules are trained, the approach can be lifted onto other text-to-video foundation models.
  • The novel-view extension turns an ordinary video into a multi-view video, so a clip can be re-rendered from any specified camera placed around the scene.
  • Quantitatively, the method reports lower relative rotation and translation error (RotErr 0.12, TransErr 0.58) and roughly 3.6 times more matched pixels than the image-to-video baselines, indicating that the generated views actually respect the specified camera geometry.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: if the frozen base model already supplies most 3D priors, then scaling this adapter to multi-view images with large baselines is mainly a data problem; the same module could be trained on internet videos with estimated camera poses to push toward fully in-the-wild multi-camera generation.
  • Editorial: the reported pose error measures relative geometry between generated views, not absolute scene reconstruction; the method may therefore be sufficient for consistent shots but not for metric 4D reconstruction without an additional alignment step.
  • Editorial: a direct testable extension would be to condition the cross-view attention on epipolar geometry only at large angular separations, where pure data-driven correspondence may stop being reliable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. SynCamMaster proposes a plug-and-play module on top of a frozen pre-trained text-to-video DiT to generate n synchronized videos from a single text prompt and n 6-DoF camera poses. A camera encoder injects extrinsic parameters into each Transformer block, and a per-frame cross-view self-attention layer exchanges features across views. To overcome the scarcity of multi-view video data, the authors train on 500 Unreal Engine-rendered scenes with 36 cameras each, DL3DV-10K image frames sampled from monocular videos, and general single-view videos copied across v identical cameras. The paper reports automatic-metric gains over image-to-video baselines on a 100-prompt, 4-view evaluation set, and extends the method to novel-view video synthesis from a reference video.

Significance. The plug-and-play design, hybrid data recipe, and released dataset are useful contributions. If the claimed open-domain synchronized multi-view video generation holds, the method would be valuable for virtual filming and dynamic scene synthesis, and the per-block cross-view attention with a frozen base model is an efficient recipe. The paper also states that code and dataset will be released, which is a concrete asset. However, the evidence for the central claim is currently incomplete: dynamic cross-view synchronization rests on an untested prior of the frozen base model, and the reported evaluations are point estimates on a small angle range. I therefore treat the contribution as promising but requiring substantial verification.

major comments (3)
  1. [3.2, 3.3] Equation (6) defines the multi-view synchronization module as a per-frame cross-view attention: the frame index is omitted, and the attention is over the n views at the same time step. There is no cross-view temporal interaction. Therefore any synchronized dynamics (e.g., a moving object appearing at corresponding positions in all views at every frame) must be supplied by the frozen T2V model's single-view spatio-temporal attention. The paper does not demonstrate that this prior transfers to cross-view temporal consistency, and the training data cannot provide such a signal on real scenes: the DL3DV multi-view image samples are frames from a single video at different time instants, and the general-video regularization copies the same camera across views. The only true synchronized multi-view video data are the 500 UE scenes with 70 assets. This gap is load-bearing for the 'excellent inter-view synchronization' claim and should be addressed, either by a temporal cross-view mechanism or by an explicit evaluation of cross-view motion correspondence at large viewpoint differences.
  2. [4.2, Appendix C] All quantitative claims in Tables 1, 2, and 4 are single point estimates on a 100-prompt evaluation set, with no confidence intervals, variance, or significance tests; given the considerable stochasticity of video diffusion models, this cannot support the claim of superiority 'by a large margin'. In addition, the camera-control evaluation in Appendix C only covers adjacent-camera azimuth differences of 10-30 degrees and elevation differences of 10-15 degrees, which is far smaller than the 'arbitrary viewpoints' and 'large viewpoint differences' highlighted in the introduction and qualitative figures. I ask for error bars or statistical testing, and for a camera-control and synchronization evaluation at larger angular separations; a human evaluation or an external benchmark would also strengthen the claim.
  3. [4.2] The baselines are initialized from reference frames generated by SynCamMaster itself (Fig. 5 and Section 4.2). This makes the comparison a measure of how well each image-to-video model can continue from SynCamMaster's own first frames, not a comparison of end-to-end multi-view video generation, and it may inflate the apparent advantage because the reference frames already encode SynCamMaster's camera-conditioned appearance. Please add an independent baseline construction (e.g., reference frames from a multi-view image generator or from the same first frame with known relative poses) and/or evaluate on an external multi-view video benchmark with ground-truth cameras.
minor comments (6)
  1. [3.2, Eqs. (5)-(6)] The symbol F^v_i is reused for the input of the MVS module and its output; rename the intermediate feature to avoid confusion.
  2. [3.4, Appendix C] In the sampling-strategy paragraph of Appendix C, the sentence beginning 'For multi-view image data, we first calculate...' appears to refer to multi-view video data; the repeated phrase is confusing.
  3. [Appendix B.2] The word 'amizuth' should be 'azimuth'.
  4. [4.3, Table 3] The claim of comparability with GCD (Van Hoorick et al., 2024) in novel-view video synthesis is not supported by Table 3, which lists only SynCamMaster settings; report the GCD numbers or remove the claim.
  5. [1, 2] The word 'pioneered' in the contribution list is too strong given the concurrent CVD work discussed in Section 2; consider rewording to 'first open-domain text-to-multi-video method' or similar.
  6. [4.1] The FVD-V metric is not defined precisely; the description 'FVD between video frames of different viewpoints within one scene' is ambiguous about how pairs are formed across frames and views.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multi-view synchronization module is trained with ground-truth multi-view supervision and evaluated against input poses and external metrics.

full rationale

The paper's central derivation is a standard supervised training setup, not a self-referential one. The multi-view synchronization module (Eqs. 5-6) is trained with the conditional flow-matching loss of Eq. 3 on ground-truth synchronized multi-view videos, multi-view images, and regularizing single-view videos. The camera encoder is conditioned on input extrinsic parameters, and the evaluation measures whether generated videos satisfy those poses via GIM-based RotErr/TransErr; this is a controllability check, not a closed loop in which the predicted quantity is the fitted input. The reliance on the frozen text-to-video base model's 3D consistency is an architectural assumption, but the method does not derive its synchronization capability from a citation alone: it trains cross-view attention on 500 UE-rendered synchronized multi-view scenes plus auxiliary data, and it reports external benchmark comparisons (FID, FVD, VBench). The citations to Kuaishou-affiliated prior work overlap with the authors, but they are not used as a uniqueness theorem or as a substitute for the training signal, so they are not load-bearing in a circular sense. The use of SynCamMaster-generated images as reference inputs for baseline methods is a benchmarking artifact that may affect fairness, but it is not a derivation-level reduction of a prediction to its inputs. The paper's own limitation section concedes detail inconsistencies in complex scenes, which is an honest scope statement rather than evidence of circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central contribution is trained, not derived, so the ledger records the hand-chosen training hyperparameters and the key modeling assumptions that the empirical claims rest on. No invented physical entities are introduced.

free parameters (5)
  • Data mixing probabilities = 0.6 multi-view video, 0.2 multi-view image, 0.2 general video
    Chosen by validation performance in Section 3.4 and Appendix D; the paper reports no sensitivity or variance across mixing ratios.
  • Progressive angle schedule = theta_l=0, theta_h=60 for 0-10K steps; 30-90 for 10K-20K; 60-120 for >20K
    Hand-designed schedule in Appendix C that the authors find crucial for viewpoint-following at large relative angles; the exact values are ad hoc and not derived.
  • CFG weights for novel view synthesis = sV=1.8, sT=7.5
    Selected from a four-point sweep in Table 3 using LPIPS, PSNR, and SSIM; these weights directly affect the extension results.
  • Reference latent replacement probability = 0.9
    Set by hand in Section 3.5 to strengthen video conditioning during novel view synthesis training; no ablation is reported for this value.
  • Training configuration = 50K steps, batch size 32, learning rate 1e-4, resolution 384x672
    Standard implementation choices in Section 4.1 that define the final model; they are not derived from a principle and may not transfer to other base models.
assumptions (6)
  • domain assumption The pre-trained text-to-video model already encodes sufficient 3D and motion priors for open-domain scenes.
    Invoked in Section 3.2 where MVS modules are inserted into a frozen base model; if the base model lacks these priors, cross-view attention cannot create them.
  • domain assumption Camera extrinsics normalized to one global camera and assumed constant across frames are a sufficient conditioning signal for arbitrary viewpoint synthesis.
    Stated in Section 3; the model only sees relative extrinsics and cannot represent moving cameras or per-camera intrinsics.
  • domain assumption Multi-view images sampled from DL3DV-10K transfer geometry-correspondence knowledge to video generation.
    Core to the hybrid training scheme in Section 3.4; supported by ablation Table 2, but not by a formal proof.
  • domain assumption Copying single-view videos v times with identical camera parameters provides useful regularization without harming synchronization.
    Used in Section 3.4 to add general video data; the paper reports that it improves visual quality but does not analyze failure modes.
  • standard math Rectified flow and conditional flow matching define the training objective and sampling schedule.
    Equations (1)-(4) in Section 3.1 rely on Lipman et al. and Esser et al.; this is standard background math.
  • domain assumption GIM-estimated correspondences and relative poses are accurate enough to evaluate synchronization and camera control.
    All RotErr, TransErr, and Mat. Pix. metrics in Tables 1 and 4 depend on GIM matching quality, which is not independently validated on this distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints." pith.science (2026). https://pith.science/paper/K4LGUHLM

@misc{pith2026241207760,
  author       = {Pith},
  title        = {Pith review of: SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4LGUHLM}},
  note         = {Machine review of arXiv:2412.07760}
}
read the original abstract

Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: https://jianhongbai.github.io/SynCamMaster/.

Figures

Figures reproduced from arXiv: 2412.07760 by the authors.

Figure 1
Figure 1. Examples synthesized by SynCamMaster. SynCamMaster generates multiple videos of the same dynamic scene from diverse viewpoints. Videos results are on our project page. ABSTRACT Recent advancements in video diffusion models have shown exceptional abili￾ties in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dy￾namic consis… view at source ↗
Figure 2
Figure 2. Overview of SynCamMaster. Based on a pre-trained text-to-video model, two compo￾nents are newly introduced: the camera encoder projects the normalized camera extrinsic parame￾ters into embedding space; the multi-view synchronization module, as plugged in each Transformer block, modulates inter-view features under the guidance of inter-camera relationship. Only new components are trainable, while the pre-trained text… view at source ↗
Figure 3
Figure 3. Data collection process. (a) Illustration of extracting multi-view image data from videos with camera movements, images are from DL3DV-10K (Ling et al., 2024); (b) Example of the ren￾dered multi-view videos from diverse viewpoints; (c) Utilizing general video data as regularization. 3.3 DATA COLLECTION The scarcity of multi-view video data is one of the main challenges hindering the training of multi￾view video gene… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Illustration of the rendering scene. Construction of the Multi-View Video Data Firstly, we collect 70 3D assets of humans and animals as the main subjects and select 500 dif￾ferent locations in 3D scenes as background. Secondly, we randomly sample 1-2 main sub￾jects to…
Figure 5
Figure 5. Figure 5: Comparison with state-of-the-art methods. The reference multi-view images of baseline methods (indicated in the blue box) are generated by SynCamMaster. It shows that SynCamMaster generates consistent content (e.g., the details in the red box) from different viewpoints…
Figure 6
Figure 6. Figure 6: Ablation on the joint training strategy. The captions on both sides represent the com￾position of the training set, where ”Mono. Video” refers to general monocular videos. It shows that training with the auxiliary multi-view image data and general video data significan…
Figure 8
Figure 8. Figure 8: Results of the extension on novel view video synthesis [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 7
Figure 7. Figure 7: Ablation on progressive training. The Effectiveness of Progressive Training View synchronized synthesis is particularly challenging when aiming to generate videos with large viewpoint differences since there are fewer matching features across views. To tackle this prob…
Figure 9
Figure 9. Figure 9: Overview of the base text-to-video generation model. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Comparasion on using different camera representations. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison of SynCamMaster with epipolar attention and full attention. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: More comparison with state-of-the-art methods. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: More synthesized results of SynCamMaster. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: More synthesized results of SynCamMaster. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Visualization of failure cases. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MV-Forcing: Long Multi-View Video Generation via 4D-Grounded Spatio-Temporal Self-Forcing

    cs.CV 2026-07 conditional novelty 7.0 of 10

    MV-Forcing composes temporal and view-sequential autoregression in a single diffusion model, using a recurrent 3D reconstruction model as a geometric bridge to generate arbitrarily long, multi-view consistent videos.

  2. FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A single video-diffusion framework composites both static images and dynamic footage along user-defined trajectories by transporting canonical foreground latents directly into the background latent sequence.

  3. Syn4D: A Multiview Synthetic 4D Dataset

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Syn4D supplies multiview synthetic dynamic scenes with dense geometric, tracking and pose ground truth that lets any pixel be unprojected to any time and camera.

  4. SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation

    cs.CV 2026-04 conditional novelty 6.0 of 10

    SymphoMotion jointly controls camera trajectories and depth-aware object dynamics inside one video diffusion model, supported by the new RealCOD-25K real-world paired-motion dataset.

  5. ShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Distributed video-generation agents can create a spatiotemporally consistent shared world by tiling four views, exchanging cross-agent attention, and querying a spatial memory cache.

  6. InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A training-free, near-zero-overhead inverse solver for novel-view video generation and inpainting that projects masks into continuous multi-channel latent masks and applies DDS with conjugate gradient in latent space.

  7. Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Vid-CamEdit re-synthesizes monocular videos along user-defined camera paths by conditioning a video diffusion model on 2D flows derived from estimated 3D geometry, without training on multi-view video data.

  8. UNIC: Unified In-Context Video Editing

    cs.CV 2025-06 conditional novelty 6.0 of 10

    One diffusion transformer handles ID insert, swap, delete, stylization, propagation, and re-camera control in a single model using in-context token concatenation with task-aware positional encoding and bias.

  9. Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Context-as-Memory conditions video generation on selected historical frames chosen by camera FOV overlap, improving scene consistency in long generated videos.

  10. From World Action Models to Embodied Brains: A Roadmap for Open-World Physical Intelligence

    cs.RO 2026-07 conditional novelty 4.0 of 10

    Physical intelligence needs an embodied brain that reasons over interventions and emits capability requests, grounded by a physical harness and shared experience contracts rather than direct actuator policies.

Reference graph

Works this paper leans on

70 extracted references · 26 canonical work pages · cited by 10 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Vd3d: Taming large video diffusion transformers for 3d camera control

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407.12781, 2024

    arXiv 2024

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  4. [4]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18392--18402, 2023

    2023

  5. [5]

    Generative novel view synthesis with 3d-aware diffusion models

    Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4217--4229, 2023

    work page 2023

  6. [6]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7310--7320, 2024

    work page 2024

  7. [7]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13142--13153, 2023

    2023

  8. [8]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \" u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024

    work page 2024

Show all 70 references
  1. [9]

    3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation

    Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. arXiv preprint, 2024

    2024

  2. [10]

    Cat3d: Create anything in 3d with multi-view diffusion models

    Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314, 2024

    2024 arXiv

  3. [11]

    Gen-3. Gen-3. Gen-3. Accessed Sept.30, 2024 [Online] https://runwayml.com/research/introducing-gen-3-alpha, 2024. URL https://runwayml.com/research/introducing-gen-3-alpha

    2024

  4. [12]

    Emu video: Factorizing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023

    2023 arXiv

  5. [13]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933, 2023 a

    2023 arXiv

  6. [14]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023 b

    2023 arXiv

  7. [15]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024

    2024 arXiv

  8. [16]

    Epipolar transformers

    Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp.\ 7779--7788, 2020

    2020

  9. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  10. [18]

    o llein, Alja z Bo z i c , Norman M \

    Lukas H \"o llein, Alja z Bo z i c , Norman M \"u ller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollh \"o fer, and Matthias Nie ner. Viewdiff: 3d-consistent image generation with text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Visi...

    2024

  11. [19]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  12. [20]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  13. [21]

    Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36 0 (7): 0 1325--1339, 2013

    2013

  14. [22]

    Videobooth: Diffusion-based video generation with image prompts

    Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6689--6700, 2024

    2024

  15. [23]

    Lvsm: A large view synthesis model with minimal 3d inductive bias

    Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint arXiv:2410.17242, 2024

    2024 arXiv

  16. [24]

    Panoptic studio: A massively multiview system for social motion capture

    Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE international conference on computer vision, pp.\ 3334--3342, 2015

    2015

  17. [25]

    Spad: Spatially aware multi-view diffusers

    Yash Kant, Aliaksandr Siarohin, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, and Igor Gilitschenski. Spad: Spatially aware multi-view diffusers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  18. [26]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023

    2023 arXiv

  19. [27]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014

    2014

  20. [28]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023

    2023

  21. [29]

    Kling. Kling. Kling. Accessed Sept.30, 2024 [Online] https://kling.kuaishou.com/en, 2024. URL https://kling.kuaishou.com/en

    2024

  22. [30]

    Collaborative video diffusion: Consistent multi-video generation with camera control

    Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. arXiv preprint arXiv:2405.17414, 2024

    2024 arXiv

  23. [31]

    Vivid-zoo: Multi-view video generation with diffusion model

    Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai, Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid-zoo: Multi-view video generation with diffusion model. arXiv preprint:2406.08659, 2024

    2024 arXiv

  24. [32]

    Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models

    Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. arXiv preprint arXiv:2405.16645, 2024

    2024 arXiv

  25. [33]

    Efficient neural radiance fields for interactive free-viewpoint video

    Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia Conference Proceedings, 2022

    2022

  26. [34]

    Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp...

    2024

  27. [35]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben - Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023

    2023

  28. [36]

    Stylecrafter: Enhancing stylized text-to-video generation with style adapter

    Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video generation with style adapter. arXiv preprint arXiv:2312.00330, 2023 a

    2023 arXiv

  29. [37]

    Zero-1-to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9298--9309, 2023 b

    2023

  30. [38]

    Syncdreamer: Generating multiview-consistent images from a single-view image

    Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023 c

    2023 arXiv

  31. [39]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan - Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024 a

    2024 arXiv

  32. [40]

    Magic-me: Identity-specific video customized diffusion

    Ze Ma, Daquan Zhou, Chun-Hsiao Yeh, Xue-She Wang, Xiuyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-specific video customized diffusion. arXiv preprint arXiv:2402.09368, 2024 b

    2024 arXiv

  33. [41]

    Snap video: Scaled spatiotemporal transformers for text-to-video synthesis

    Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In Proceedings of the IEEE/CVF Conference...

    2024

  34. [42]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  35. [43]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  36. [44]

    Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction

    Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference on computer visi...

    2021

  37. [45]

    An introduction to Unreal engine 4

    Andrew Sanders. An introduction to Unreal engine 4. AK Peters/CRC Press, 2016

    2016

  38. [46]

    Ntu rgb+ d: A large scale dataset for 3d human activity analysis

    Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 1010--1019, 2016

    2016

  39. [47]

    Gim: Learning generalizable image matcher from internet videos

    Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In The Twelfth International Conference on Learning Representations, 2024

    2024

  40. [48]

    Mvdream: Multi-view diffusion for 3d generation

    Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512, 2023

    2023 arXiv

  41. [49]

    Sora. Sora. Sora. Accessed Sept.30, 2024 [Online] https://openai.com/index/video-generation-models-as-world-simulators/, 2024. URL https://openai.com/index/video-generation-models-as-world-simulators/

    2024

  42. [50]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha \"e l Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. Openreview, 2019

    2019

  43. [51]

    Generative camera dolly: Extreme monocular dynamic novel view synthesis

    Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. arXiv preprint arXiv:2405.14868, 2024

    2024 arXiv

  44. [52]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023 a

    2023 arXiv

  45. [53]

    Lavie: High-quality video generation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023 b

    2023 arXiv

  46. [54]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp.\ 1--11, 2024

    2024

  47. [55]

    4d gaussian splatting for real-time dynamic scene rendering

    Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20310--20320, 2024

    2024

  48. [56]

    Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency

    Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency. arXiv preprint arXiv:2407.17470, 2024

    2024 arXiv

  49. [57]

    Make-your-video: Customized video generation using textual and structural guidance

    Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, et al. Make-your-video: Customized video generation using textual and structural guidance. IEEE Transactions on Visualization and Computer Graphics, 2024

    2024

  50. [58]

    Camco: Camera-controllable 3d-consistent image-to-video generation

    Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509, 2024

    2024 arXiv

  51. [59]

    Direct-a-video: Customized video generation with user-directed camera movement and object motion

    Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pp.\ 1--12, 2024 a

    2024

  52. [60]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pp.\ 1--11, 2023

    2023

  53. [61]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models ...

    2024 arXiv

  54. [62]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023

    2023 arXiv

  55. [63]

    Mvimgnet: A large-scale dataset of multi-view images

    Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognit...

    2023

  56. [64]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023 a

    2023 arXiv

  57. [65]

    4diffusion: Multi-view video diffusion model for 4d generation

    Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-view video diffusion model for 4d generation. arXiv preprint arXiv:2405.20674, 2024

    2024 arXiv

  58. [66]

    Controlvideo: Training-free controllable text-to-video generation

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023 b

    2023 arXiv

  59. [67]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817, 2018

    2018 arXiv

  60. [68]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  61. [69]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  62. [70]

    Mono. Video

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1930

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.