Pith. sign in

REVIEW 4 major objections 6 minor 13 cited by

Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conditioning video diffusion on 3D point trajectories unifies camera, motion, and mesh control in one model.

desk verdict A genuinely new 3D control signal for video diffusion, with a solid core ablation but quantitative evaluation that needs strengthening before the headline numbers are trustworthy. read the letter →

arxiv 2501.03847 v2 pith:7L4FNR6R submitted 2025-01-07 cs.CV cs.AIcs.GR

Zekai Gu , Rui Yan , Jiahao Lu , Peng Li , Zhiyang Dou , Chenyang Si , Zhen Dong , Qifeng Liu
show 4 more authors
Cheng Lin Ziwei Liu Wenping Wang Yuan Liu
This is my paper · ORCID
classification cs.CVcs.AIcs.GR
keywords videodiffusion3Dtrackingcontrollablegenerationmotiontransfercameracontrolmesh-to-videoobjectmanipulationtemporalconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the reason video generation is hard to control is that the controls are 2D, while video is a 2D render of a moving 3D scene. It proposes conditioning an image-to-video diffusion model on a 3D tracking video: a video whose pixels are colored 3D points that keep their colors across frames as they move. Because each point keeps its color, the same 3D location stays identifiable over time, which the authors say is what makes generated videos temporally consistent and what allows one architecture to cover mesh-to-video generation, camera control, motion transfer, and object manipulation. The model is a fine-tuned CogVideoX with a frozen denoiser and a trainable condition branch, trained on fewer than 10k videos in three days.

What carries the argument

The central object is the 3D tracking video, a temporal sequence of frames in which each pixel belongs to a 3D point whose color is fixed by its first-frame coordinate, so identical points remain color-anchored across time. The defining property is that this color anchoring links frames: unlike a depth map, which describes each frame's structure independently, a tracking video tells the model which 3D point is which at every time step. The conditioning is injected by a trainable copy of the first 18 blocks of the denoising DiT, whose per-block features are added to the frozen denoiser through zero-initialized linear layers, following the ControlNet recipe.

What would settle it

Recolor each point in the tracking video per frame so that colors carry no cross-frame identity while leaving the trajectories unchanged; if DaS still produces temporally consistent video, the claimed color-anchoring mechanism is not what drives the effect.

Watch

Extended reading notes

Core claim

The central claim is that a 3D tracking video is a sufficient control signal for versatile, precise video generation. DaS colorizes dynamic 3D points by their normalized coordinates in the first frame's camera space, renders their trajectories into a tracking video, and injects that video into a latent diffusion transformer through a ControlNet-style trainable copy of the first 18 DiT blocks with zero-initialized linear layers. The frozen base model acts as a shader that paints appearance onto the moving points. The authors show this single mechanism handles four control tasks, and they report that it beats depth-map conditioning on reconstruction metrics (PSNR, SSIM, LPIPS, FVD) and beats MotionCtrl and CameraCtrl on camera-trajectory error and TokenFlow and CCEdit on motion-transfer CLIP scores.

Load-bearing premise

The load-bearing premise is that the 3D tracking videos fed to the model are accurate, dense, and temporally consistent; if the upstream tracker or monocular depth estimator misplaces points, the model either ignores the control or invents implausible content.

Editorial extensions

If this is right

  • Camera control becomes a matter of projecting estimated 3D points onto a desired camera path, with no camera embedding or ray representation needed.
  • Motion transfer reduces to pairing a repainted first frame with the source video's 3D tracking video, so the same motion can be restyled without per-video optimization.
  • Mesh-to-video generation works by rendering an animated mesh into a tracking video, letting a prompt-driven first frame determine the appearance.
  • Object manipulation works by editing the point cloud of a segmented object and rendering the edited points back into a tracking video.
  • Because the base denoiser is frozen and only the condition branch is trained, supporting a new control task requires only constructing a tracking video, not retraining a specialized model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the color-anchoring mechanism suggests that tracking-video conditioning could be ported to other image-to-video backbones, since it only adds a condition branch to a frozen denoiser.
  • A testable extension of the paper's reliance on upstream estimators is to feed DaS synthetic tracking videos with controlled noise added to point positions, then measure how camera-trajectory error and object-manipulation fidelity degrade as tracker accuracy drops.
  • The paper implicitly claims that a training set of human-centric Mixamo meshes plus MiraData real videos transfers to arbitrary open-domain content; one could test this by evaluating on scenes with no human or articulated objects.
  • The authors note that generating 3D tracking videos with a diffusion model is a future direction; if that works, the input requirement would shift from existing videos or meshes to a text or image prompt alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Diffusion as Shader (DaS), an image-to-video diffusion model conditioned on 3D tracking videos—videos of colored 3D point trajectories. The method freezes a pretrained CogVideoX denoising DiT and fine-tunes a trainable copy of its first 18 blocks, injecting condition features through zero-initialized linear layers. The paper claims that this single 3D control modality unifies four video-generation-control tasks: animating meshes to videos, camera control, motion transfer, and object manipulation, while also improving temporal consistency relative to depth-based control. Training uses fewer than 10k videos and 3 days on 8 H800 GPUs. The quantitative evaluation covers camera control (Table 1), motion transfer (Table 2), and an ablation comparing depth versus 3D tracking videos and varying track density (Table 3); mesh-to-video and object manipulation are evaluated qualitatively.

Significance. The central hypothesis—that a shared 3D tracking-video representation can serve as a unified, precise control signal for diverse video-generation tasks—is well-motivated and potentially impactful for content creation. The architecture is simple and data-efficient, and the controlled ablation in Table 3 (depth versus 3D tracking on the same architecture) is a genuine strength: it directly supports the claim that 3D tracking videos confer an advantage over per-frame depth for temporal consistency. The qualitative results for mesh-to-video and object manipulation are also compelling. However, the quantitative evidence for the headline claims of precise camera control and faithful motion transfer is currently fragile; the metrics used either have unsound components (camera pose estimation from a mis-cited SIFT) or do not measure the claimed property (CLIP-based motion-transfer metrics). If these evaluation issues are fixed and the method still wins, the paper would make a solid contribution.

major comments (4)
  1. [§4.1, Table 1] The camera-control metric is not reliable as reported. Pose errors are computed by estimating relative camera poses from generated videos with SIFT, but the cited reference (Ng & Henikoff 2003) is a protein-sequence alignment tool, not an image-feature SIFT, and no implementation details, feature-matching success rates, or RANSAC settings are given. Monocular relative translation is recoverable only up to scale; comparing normalized translation vectors therefore measures direction noise rather than control accuracy. The paper should either use a metric based on known renderings of a 3D scene (e.g., rendered mesh videos) where ground-truth camera pose is known, or provide a validated pose-estimation pipeline with reported success rates and account for scale ambiguity. Without this, the quantitative superiority over MotionCtrl and CameraCtrl in Table 1 is not established.
  2. [§4.2, Table 2] The motion-transfer metrics do not measure motion fidelity. Tex-Ali (CLIP text alignment) and Tem-Con (adjacent-frame CLIP similarity) can both be high for a video that ignores the source motion entirely: a static, text-aligned video achieves perfect Tem-Con and can achieve high Tex-Ali. The claim that DaS 'accurately transfers the video motion' is therefore unsupported by Table 2. The paper should add a motion-fidelity metric, such as point-trajectory similarity (e.g., comparing SpatialTracker or optical-flow tracks of the source video with those of the generated video), or a user study evaluating whether the source motion was transferred.
  3. [§4.5.1, Table 3] The depth baseline in Table 3 uses per-frame Depth Pro to construct the depth video, so temporal inconsistency is built into the comparison by construction. To isolate the benefit of 3D tracking videos' cross-frame association, the control condition should use a video-consistent depth estimator such as Align3R (already cited in the paper as [Lu et al. 2024]) or should enforce temporal consistency on the depth maps. Additionally, no error bars or significance tests are reported for PSNR/SSIM/LPIPS/FVD, and the default of 4900 tracks was selected from the same 50-video validation set used to report the ablation numbers, which mildly inflates the chosen configuration's performance. The paper should report variance over multiple seeds or dataset splits and ideally perform model-selection on a separate set.
  4. [§3.3 and §4.5] The paper states that 'we randomly selected 50 videos from the validation split' for the analysis in §4.5, and then uses this same set to choose the default number of 3D tracking points (4900). This is a form of selection-on-validation: the reported numbers for the 4900-track configuration are on data that influenced the choice, and the gap between 4900 and 8100 tracks may be partly due to tuning. This should be acknowledged, and ideally the final configuration should be re-evaluated on a held-out set.
minor comments (6)
  1. [Figure 10 caption] The caption contains a typo: 'CogVdieoX' should be 'CogVideoX'.
  2. [Table 2 and §4.2 text] The table uses 'Tex-Ali' while the text uses 'Text-Ali'; please unify the terminology.
  3. [§4.5.2] The sentence 'the visual qualities of 2500, 4900, and 8100 tracking points are very similar to each other' is contradicted by the FVD column (566.4, 551.3, 599.0) and, for 2500, by LPIPS (0.263 vs 0.261). Please make the claim consistent with the reported numbers or provide statistical support for 'similar'.
  4. [§4.1, Eq. (1) and (2)] The equations for RotErr and TransErr are correct in spirit but should specify how the ground-truth camera poses are obtained for the 'Large Movement' cases (e.g., generated from a known trajectory) and whether the estimated poses are aligned to the ground-truth coordinate frame.
  5. [§2.2] The related-work paragraph on CameraCtrl is grammatically incomplete: 'CameraCtrl [He et al. 2024b] adopts Plücker embeddings ... enabling the ViewCrafter [Yu et al. 2024] employs...' This should be rewritten.
  6. [§5, Figure 11] The failure-case discussion is honest, but it would be useful to quantify how often these failures occur, e.g., the fraction of test videos where the tracking video is 'incompatible' with the input image.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DaS is an empirical conditioning architecture evaluated against external baselines; the reported ablation hyperparameter choice is explicit validation selection, not a fitted prediction.

full rationale

The paper's central claim is that conditioning an image-to-video diffusion model on 3D tracking videos enables versatile control. This is an empirical system paper, not a derivation, and the load-bearing comparisons are against external baselines (MotionCtrl, CameraCtrl, TokenFlow, CCEdit, CHAMP). The 3D tracking video is constructed from explicit 3D points and injected via a ControlNet-style condition DiT; no equation or definition reduces the claimed control capability to the input by construction. The only near-concern is the choice of 4900 tracking points in Sec. 4.5.2, selected from the same 50-video validation set used for the Table 3 ablation; this is a disclosed hyperparameter choice, not a fitted parameter renamed as a prediction, and it does not drive the external quantitative comparisons. The depth-vs-tracking comparison in Table 3 is somewhat favorable by design because the depth baseline uses per-frame Depth Pro without cross-frame association, but that is an experimental fairness issue, not circularity. The camera-control metric cites a protein-sequence SIFT reference and the motion-transfer metric does not directly measure motion fidelity; these are correctness/evidence-quality limitations, not circular reductions. The paper's reliance on upstream estimators (SpatialTracker, Depth Pro, MoGE, SAM) is an external dependency acknowledged in the limitations section, not a self-referential argument. No self-citation is load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The method rests on standard diffusion and ControlNet machinery plus two off-the-shelf 3D estimators (SpatialTracker and monocular depth). The only numerical parameter the central claim really depends on is the track density, which was chosen from a small validation set. The main risky assumption is that the upstream 3D estimates are accurate enough to serve as ground-truth control signals.

free parameters (1)
  • Default number of 3D tracking points = 4900
    Selected from an ablation on 50 validation videos (Table 3) where 4900 gave the best PSNR and SSIM; the paper notes 2500-8100 are visually similar, so the central claim is not sensitive to this choice, but the reported default is tuned on the evaluation set.
assumptions (5)
  • domain assumption Videos are 2D renderings of dynamic 3D content, so 3D control signals subsume 2D controls.
    Stated in Sec 1 as the key insight; not proven, but it is the premise that motivates the 3D tracking video representation.
  • domain assumption SpatialTracker produces 3D trajectories accurate enough to train and condition the model.
    Used in Sec 3.3 for real-world training data and in Sec 3.4.4 for motion transfer; tracking errors lead to the failure cases acknowledged in Sec 5.
  • domain assumption Monocular depth estimates (Depth Pro, MoGE) are metric-scale and accurate enough for back-projecting pixels to 3D points and re-projecting them onto novel camera paths.
    Used in Sec 3.4 for camera control and object manipulation; any depth scale error breaks the constructed 3D tracking video.
  • ad hoc to paper The condition DiT (first 18 of 42 blocks) with zero-init linear injections is sufficient to convey the 3D tracking signal to the frozen denoising DiT.
    The architecture in Sec 3.3 is chosen without an ablation over the number of copied blocks; the paper does not justify 18 blocks.
  • standard math Standard pinhole camera projection and coordinate-to-RGB colorization are used to render 3D tracking videos.
    Invoked in Sec 3.3 for constructing the control input; these are standard graphics operations.
invented entities (1)
  • 3D tracking video (colored 3D point trajectories as an RGB video)
    purpose: A control signal that encodes 3D scene structure and motion for the video diffusion model, unifying camera, object, and motion control.
    The representation is new to this paper; its utility is demonstrated only by the paper's own experiments, and no external benchmark or independent implementation validates it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control." pith.science (2026). https://pith.science/paper/7L4FNR6R

@misc{pith2026250103847,
  author       = {Pith},
  title        = {Pith review of: Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7L4FNR6R}},
  note         = {Machine review of arXiv:2501.03847}
}
read the original abstract

Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process, such as camera manipulation or content editing, remains a significant challenge. Existing methods for controlled video generation are typically limited to a single control type, lacking the flexibility to handle diverse control demands. In this paper, we introduce Diffusion as Shader (DaS), a novel approach that supports multiple video control tasks within a unified architecture. Our key insight is that achieving versatile video control necessitates leveraging 3D control signals, as videos are fundamentally 2D renderings of dynamic 3D content. Unlike prior methods limited to 2D control signals, DaS leverages 3D tracking videos as control inputs, making the video diffusion process inherently 3D-aware. This innovation allows DaS to achieve a wide range of video controls by simply manipulating the 3D tracking videos. A further advantage of using 3D tracking videos is their ability to effectively link frames, significantly enhancing the temporal consistency of the generated videos. With just 3 days of fine-tuning on 8 H800 GPUs using less than 10k videos, DaS demonstrates strong control capabilities across diverse tasks, including mesh-to-video generation, camera control, motion transfer, and object manipulation.

Figures

Figures reproduced from arXiv: 2501.03847 by the authors.

Figure 1
Figure 1. Diffusion as Shader (DaS) is (a) a 3D-aware video diffusion method enabling versatile video control tasks including (b) animating meshes to video generation, (c) motion transfer, (d) camera control, and (e) object manipulation. Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process—such as cam… view at source ↗
Figure 2
Figure 2. Architecture of DaS. (a) We colorize dynamic 3D points according to their coordinates to get (b) a 3D tracking video. (c) The input image and the 3D tracking video are processed by (d) a transformer-based latent diffusion with a variational autoencoder (VAE). The 3D tracking video is processed by a trainable copy of the denoising DiT and zero linear layers are used to inject the condition features from 3D tracking v… view at source ↗
Figure 3
Figure 3. 3D tracking video generation in (a) object manipulation, (b) animating mesh to video generation, (c) camera control, and (d) motion transfer. strategy to get an effective batch size of 64. The training takes 3 days on 8 H800 GPUs. 3.4 Video generation control In this section, we describe how to utilize DaS for the following controllable video generation. 3.4.1 Object manipulation. DaS can generate a video to manipul… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Qualitative results of DaS on the camera control task. We show 4 trajectories (left, right, up, down) with large movements. camera poses from the generated videos and the input ground-truth camera poses using rotation errors and translation errors. Specifi￾cally, for e…
Figure 5
Figure 5. Figure 5: Qualitative comparison on motion transfer between our method, CCEdit [Feng et al. 2024b], and TokenFlow [Geyer et al. 2023b]. Method Tex-Ali ↑ Tem-Con ↑ CCEdit 16.9 0.932 Tokenflow 31.9 0.956 Ours 32.6 0.971 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on motion transfer of our method. Mesh Style1 Style2 Style3 Style4 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: More results of the animating mesh to video generation task. Our method enables the generation of different styles from the same mesh [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison on the animating mesh to video task between our method and CHAMP [Zhu et al. 2024]. Input Image Translation Input Image Rotation [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Qualitative results of our method on the object manipulation task. The top part shows the results of translation while the bottom part shows the results of rotating the object [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Generated videos using depth maps or 3D tracking videos as control signals. Our 3D tracking videos provide better quality on the cross-frame consistency for video generation than depth maps. 4.5 Analysis We conduct analysis on the choice of 3D control signals, i.e. de…
Figure 11
Figure 11. Figure 11: Failure cases. (Top) Incompatible tracking video. When a tracking video that does not correspond to the structures of the input image is provided, DaS will generate a video with a scene transition to a compatible new scene. (Bottom) Out of tracking range. For regions …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation

    cs.CV 2025-02 conditional novelty 7.0 of 10

    Users control a text-to-video model by arranging 3D bounding boxes and camera motion, and the system renders depth maps that steer the diffusion model to generate matching object and camera motion.

  2. Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Tracking plus world-position maps in a neural G-buffer outperform depth as a geometric condition for reference-guided video diffusion rendering on a 68-clip synthetic benchmark.

  3. Motion4Motion: Motion Transfer Across Subjects at Inference

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training-free motion transfer across species works by extracting source motion flows, matching semantic points, and injecting them into DiT self-attention via TransPE positional padding.

  4. PostCam: Camera-Controllable Novel-View Video Generation with Query-Shared Cross-Attention

    cs.CV 2025-11 conditional novelty 6.0 of 10

    PostCam generates new videos from a reference video along user-specified camera trajectories using a query-shared cross-attention that fuses pose data and rendered frames, improving control precision and detail preservation.

  5. ANYPORTAL: Zero-Shot Consistent Video Background Replacement

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A training-free video background replacement pipeline that keeps the foreground pixel-consistent by projecting refined latents through a deterministic reparameterization.

  6. O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editing

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A video editor trained on randomly distorted objects, then steered by adaptive noise at inference, is claimed to surpass dedicated and unified editors across eight tasks with far less training.

  7. Discovering and using Spelke segments

    cs.CV 2025-07 conditional novelty 6.0 of 10

    SpelkeNet, a self-supervised video world model, discovers Spelke segments in static images by aggregating motion correlations across imagined pokes.

  8. SpatialTrackerV2: 3D Point Tracking Made Easy

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A single feed-forward model jointly estimates video depth, camera poses, and 3D point trajectories from monocular video, setting a new state of the art on TAPVid-3D.

  9. EX-4D: EXtreme Viewpoint 4D Video Synthesis via Depth Watertight Mesh

    cs.CV 2025-06 conditional novelty 6.0 of 10

    EX-4D uses a depth watertight mesh and simulated occlusion masks to condition a video diffusion model for extreme-viewpoint 4D video synthesis from monocular input.

  10. Video World Models with Long-term Spatial Memory

    cs.CV 2025-06 conditional novelty 6.0 of 10

    An autoregressive video world model with a persistent static point-cloud spatial memory and sparse episodic keyframes improves revisit consistency over point-cloud-conditioned baselines.

  11. IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

    cs.CV 2025-06 reject novelty 6.0 of 10

    A diffusion video model that jointly uses HDR lighting, relit frames, and 3D point tracks to relight videos from text prompts.

  12. EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    EPiC trains a 30M-parameter visibility-aware ControlNet on mask-based anchor videos from 5,000 in-the-wild videos and 500 steps, reaching SOTA camera accuracy on RealEstate10K and MiraData.

  13. Dynamic View Synthesis as an Inverse Problem

    cs.CV 2025-06 reject novelty 3.0 of 10

    Dynamic view synthesis from a monocular video is achieved by redesigning the noise initialization of a pretrained video diffusion model using a recursive interpolation and a stochastic latent modulation.

Reference graph

Works this paper leans on

11 extracted references · 2 canonical work pages · cited by 13 Pith papers

  1. [2]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models. In ECCV. 330–348. Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. 2023. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023). Hao He, Yingh...

    arXiv 2023

  2. [4]

    arXiv preprint arXiv:2412.06016 (2024)

    Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. arXiv preprint arXiv:2412.06016 (2024). Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. 2024. MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions. arXiv:2407.06358 [cs.CV] https:...

    arXiv 2024

  3. [6]

    arXiv preprint arXiv:2405.13865 (2024)

    ReVideo: Remake a Video with Motion and Content Control. arXiv preprint arXiv:2405.13865 (2024). Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B Lindell. 2024. Sg-i2v: Self-guided trajectory control in image-to-video generation. arXiv preprint arXiv:2411.04989 (2024). Pauline C Ng and Steven Henikoff. 2003. SIFT: Pred...

    arXiv 2024

  4. [8]

    High-resolution image synthesis with latent diffusion models. In CVPR. Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. 2024. Motion- i2v: Consistent and controllable image-to-video generation with explicit motion modeling. In SIGGRAPH. Vincent Sitzmann, Semon Rezchik...

    work page 2024

  5. [10]

    arXiv preprint arXiv:2309.08523 (2023)

    Breathing new life into 3d assets with generative repainting. arXiv preprint arXiv:2309.08523 (2023). Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2024d. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Process- ing Systems 36...

    arXiv 2023

  6. [2018]

    arXiv:1801.03924 [cs.CV] https://arxiv.org/abs/1801.03924 Tingyang Zhang, Chen Wang, Zhiyang Dou, Jiahui Lei Qingzhe Gao, Baoquan Chen, and Lingjie Liu

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv:1801.03924 [cs.CV] https://arxiv.org/abs/1801.03924 Tingyang Zhang, Chen Wang, Zhiyang Dou, Jiahui Lei Qingzhe Gao, Baoquan Chen, and Lingjie Liu. 2025. ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking. arXiv preprint arxiv:2501.03220 (2025). Guangcon...

    arXiv 2025

  7. [2020]

    IEEE transactions on pattern analysis and machine intelligence 44, 3 (2020), 1623–1637

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence 44, 3 (2020), 1623–1637. Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. 2023. Texture: Text-guided texturing of 3d shapes. In ACM SIGGRAPH 2023 conference proceedings....

    work page 2020

  8. [2021]

    Advances in Neural Information Processing Systems 34 (2021), 19313– 19325

    Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems 34 (2021), 19313– 19325. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020). Yao Teng, Enze Xie, Yue Wu, Haoyu Han, Zhenguo Li, and Xihui Liu. 2...

    arXiv 2021

Show all 11 references
  1. [2022]

    arXiv:2104.08718 [cs.CV] https://arxiv.org/abs/2104.08718 Jonathan Ho, Ajay Jain, and Pieter Abbeel

    CLIPScore: A Reference-free Evaluation Metric for Image Captioning. arXiv:2104.08718 [cs.CV] https://arxiv.org/abs/2104.08718 Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. NeurIPS (2020). Jonathan Ho, Tim Salimans, Alexey Gritsenko,...

    2020 arXiv

  2. [2023]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision

    Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026. Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, et al . 2024. Hunyuan- Video: A Systematic Framework For Large Video Generative Models. arXiv preprint arXiv:2412....

    2024 arXiv

  3. [2024]

    arXiv preprint arXiv:2412.02700 (2024)

    Motion Prompting: Controlling Video Generation with Motion Trajectories. arXiv preprint arXiv:2412.02700 (2024). Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2023a. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.1037...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.