Pith. sign in

REVIEW 3 major objections 5 minor 68 references

TransPixeler: Advancing Text-to-Video Generation with Transparency

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TransPixeler adapts pretrained video models to output RGBA video with transparency.

desk verdict Plausible and potentially important method for text-to-RGBA video, but the paper omits the alpha decoder and ships weak quantitative evidence; the core idea is worth refereeing if the gap is closed. read the letter →

arxiv 2501.03006 v2 pith:J7EFOIZB submitted 2025-01-06 cs.CV

Luozhou Wang , Yijun Li , Zhifei Chen , Jui-Hsien Wang , Zhifei Zhang , He Zhang , Zhe Lin , Yingcong Chen This is my paper · ORCID
classification cs.CV
keywords text-to-videogenerationRGBAvideoalphachanneltransparencydiffusiontransformerLoRAfine-tuningattentionmechanismmatting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TransPixeler aims to make a pretrained text-to-video diffusion transformer produce RGBA video — RGB pixels plus a per-pixel transparency (alpha) channel — without training a new model from scratch. The authors double the token sequence, treat the second half as alpha tokens, and adapt only a small set of parameters: a low-rank (LoRA) branch on alpha tokens, a learnable domain embedding, and an attention mask that blocks text-to-alpha attention but keeps RGB-to-alpha attention. The result, they report, is joint generation of foreground and matte that stay aligned in motion, trained on only about 484 RGBA videos while preserving the base model's RGB quality. For VFX work, compositing smoke, reflections, and effects into footage directly from a text prompt would become far simpler than the current two-step generate-then-matte route.

What carries the argument

The mechanism is a reorganized self-attention inside the DiT block, viewed as a 3×3 grouped matrix over Text, RGB, and Alpha tokens. The original Text–RGB block is kept untouched; Text-attend-to-Alpha is masked out with $M^* = -\infty$ for that block; RGB-attend-to-Alpha is preserved so $\alpha$ features can influence RGB outputs; and a low-rank LoRA branch is applied only to $\alpha$-token key/query/value projections, with strength controlled by a residual factor $\gamma$. Alpha tokens share the RGB positional encoding and are distinguished only by a zero-initialized learnable domain embedding, which makes the two halves start nearly identical and accelerates convergence. This combination converts a text-to-RGB model into a text-to-RGBA model with minimal trainable changes.

What would settle it

Reproduce the pipeline with the released code and inspect whether the second half of the doubled latent sequence is passed through the same frozen RGB VAE decoder that the base model uses. If it is, the output of that decoder is an RGB image, not a single-channel alpha matte, so the method as written cannot produce RGBA video. If the code instead contains an additional or modified decoder for alpha, then the paper omits a load-bearing component.

Watch

Extended reading notes

Core claim

The paper's central claim is that a frozen pretrained DiT text-to-video model can be extended to jointly generate RGB and alpha video by appending alpha tokens, sharing positional embeddings between the RGB and alpha halves, adding a zero-initialized learnable domain embedding, and applying LoRA only to the alpha-token projections. The authors identify RGB-attend-to-Alpha attention as the essential interaction that lets alpha information refine RGB tokens for alignment, and Text-attend-to-Alpha attention as harmful; they remove the latter with an attention mask. After 5,000 iterations of fine-tuning on a 484-video RGBA dataset, the model produces RGBA videos whose RGB and alpha motion align while the base model's RGB generation quality is retained. The paper supports this with qualitative comparisons to generation-then-prediction pipelines and a joint-generation baseline, plus a user study favoring the proposed method on alignment and motion quality.

Load-bearing premise

The method assumes the alpha half of the doubled latent sequence can be decoded into a valid single-channel alpha video, but the paper never describes how the pretrained RGB video decoder accomplishes that, leaving the pipeline as written incomplete at the decoding step.

Editorial extensions

If this is right

  • A pretrained text-to-video model can become an RGBA generator with a small LoRA-rank-128 fine-tune, avoiding the need for a large RGBA video corpus.
  • Joint generation with alpha-to-RGB feedback aligns the foreground with its matte better than generate-then-predict pipelines, which pass information only from RGB to alpha.
  • Blocking text-to-alpha attention protects the base model's RGB output; keeping RGB-to-alpha attention is what makes alignment possible.
  • The same extension works for image-to-video, animating a single input image while generating or propagating its alpha.
  • The approach carries over to DiT video backbones beyond the primary testbed, as the paper validates it on a second, higher-frame-rate variant of the base architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The token-doubling and attention-rectification recipe is modality-agnostic: the same masked partial LoRA could extend a pretrained video model to output other per-pixel channels (depth, normals, segmentation) with comparably little data.
  • Because the alpha tokens start as near-copies of the RGB tokens (shared position, zero domain embedding), the model is effectively learning a residual or refinement map from RGB to matte-rich features, which could be exploited to shrink the 5,000-iteration fine-tune.
  • A direct test of the paper's causal story: in the released code, toggle only the RGB-attend-to-Alpha attention block on and off; if the paper's mechanism is correct, the RGB-to-alpha optical-flow difference should change substantially while RGB quality (FVD) stays roughly constant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TransPixeler, a method to extend pretrained DiT-based text-to-video models (specifically CogVideoX) to jointly generate RGB and alpha (RGBA) video. The approach doubles the video token sequence, adds alpha-specific tokens with shared positional embeddings, a zero-initialized learnable domain embedding, and a masked partial LoRA applied only to alpha-token projections. The authors claim this preserves the original RGB generation quality while producing well-aligned alpha channels, despite training on only 484 RGBA videos from VideoMatte240K. Experiments include qualitative comparisons against matting and generation-then-prediction baselines, a user study, and quantitative metrics (FVD and optical-flow difference) on 80 self-generated videos.

Significance. If the method works as described, it would be a practically valuable contribution: enabling text-to-RGBA video generation from a pretrained model with very limited training data, with potential applications in VFX and interactive media. The paper's strengths include a clear analysis of attention interactions (Text/RGB/Alpha), a carefully designed attention mask, and the release of code. However, the significance is currently undermined by an incomplete description of the alpha latent decoding path, which is essential for the method to be reproducible and for the reported results to be meaningful.

major comments (3)
  1. [Section 3.2, after Eq. (4)] The paper states that the second half of the token sequence, x_{L+1:2L}, 'will be decoded into the corresponding alpha video,' but no alpha-capable decoder is described anywhere. The base model CogVideoX has a pretrained VAE decoder that maps latent codes to RGB images; applying that frozen decoder to the alpha latent half would produce a three-channel RGB image, not a single-channel matte. Similarly, the paper does not specify how ground-truth alpha frames are encoded to latent space for training the diffusion loss. Without an alpha encoder/decoder, the pipeline as written cannot be trained or run end-to-end, and the central claim of joint RGBA generation is not reproducible from the manuscript.
  2. [Section 4.3, Fig. 11] The quantitative evaluation uses only 80 self-generated videos, with no error bars, confidence intervals, or significance tests. FVD is computed by comparing each method's RGB outputs to those of the original RGB model, which can favor methods that closely mimic the base model even if they fail to generate meaningful alpha; flow difference is computed only on the method's own RGB and alpha outputs, so it cannot confirm that the generated alpha is correct in an absolute sense. The absence of any statistical analysis means the reported 'good balance' in Fig. 11 is not established with confidence.
  3. [Section 4.2, Table 1] The user study reports that 'Ours' wins 93.3% for RGBA alignment and 78.3% for motion quality, but with only 30 videos and 87 participants, no statistical test (e.g., a paired test or confidence interval) is provided. The claim that the method is 'obviously favored' is therefore not rigorously supported, and the comparison is only against a single baseline (LayerDiffusion + AnimateDiff).
minor comments (5)
  1. [Abstract] The abstract uses 'TransPixar' in one sentence ('TransPixar leverages a diffusion transformer') while the rest of the paper uses 'TransPixeler'; this inconsistency should be fixed.
  2. [Section 3.2, Eq. (5) and (6)] The notation for the modified positional encoding f* is confusing because it overloads the original f with a star, and the equations do not clearly define how the learnable domain embedding d is expanded from dimension D to the token dimension; a more precise definition would help reproducibility.
  3. [Section 4.3, Fig. 10 caption] The caption mentions 'e.g., reverse rocket' and 'butterflies waving their wings', but the figure shows a butterfly; the 'reverse rocket' example is not visible in the figure or described in the text, which is confusing.
  4. [Section 4, Model paragraph] The hyperparameters for the LoRA residual strength gamma (Eq. 6) and the attention mask (Eq. 7) are not reported; the paper only gives the LoRA rank (128) and training iterations. Providing these values would allow readers to replicate the experiments.
  5. [Supplementary, Section 8] The preprocessing section defines gamma = 1.1 and choke = 0.5 as 'gain' and 'choke' for color decontamination, but these symbols collide with the LoRA residual strength gamma used in the main text; using distinct names or symbols would avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical training scheme evaluated against external references, not a prediction that reduces by construction to its fitted inputs.

full rationale

The paper's central contribution is a training scheme that extends a pretrained DiT video model with alpha-specific tokens, shared positional embeddings, a learnable domain embedding, a partial LoRA, and an attention mask, trained on VideoMatte240K and evaluated on generated videos. There is no derivation of a quantitative prediction from fitted constants that is then reported as an independent result. The evaluation metrics are external or alignment-based: FVD compares the method's RGB outputs to those of the original CogVideoX model, optical-flow difference measures alignment between the method's own RGB and alpha outputs, and the user study asks humans to compare alignment and motion quality. None of these quantities is constructed from the method's fitted parameters in a way that forces the reported outcome. The cited works, including Lotus, Marigold, LayerDiffusion, and CogVideoX, function as baselines or base models rather than as load-bearing self-citations that license the central claim. One notable caveat is that Section 3.2 states the alpha half of the doubled latent sequence 'will be decoded into the corresponding alpha video' without specifying how the frozen RGB VAE decoder produces a single-channel matte, but an unspecified or missing component is a completeness and correctness concern, not a circular reduction. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. The principal borrowed assumptions are the transferability of the pretrained video model's representations and the existence of a way to decode alpha tokens, which is not documented. The free parameters are mostly training and preprocessing hyperparameters.

free parameters (6)
  • LoRA rank = 128
    Set by hand for alpha-token QKV projections; affects capacity of adaptation.
  • LoRA residual strength gamma = not stated
    Introduced in Eq. 6 to control residual strength; no value or schedule is reported.
  • Domain embedding = learnable, initialized to zero
    Zero-initialized learnable vector of shape 1xD used to distinguish alpha tokens from RGB tokens.
  • Preprocessing gain gamma = 1.1
    Used in color decontamination to refine alpha masks before training.
  • Preprocessing choke chi = 0.5
    Used to adjust mask edge sharpness in color decontamination.
  • Background blur kernel size = 201
    Large Gaussian blur applied to first frame for training background simplification.
assumptions (4)
  • domain assumption Pretrained DiT video model provides a latent space suitable for both RGB and alpha token sequences.
    The method doubles the token sequence and assumes the second half can represent alpha at the same latent scale, stated in Section 3.2.
  • domain assumption The alpha half of the latent sequence can be decoded by the existing VAE decoder (or an unspecified decoder) into a single-channel alpha map.
    Never stated explicitly; the paper simply says the doubled sequence is 'decoded into the corresponding alpha video'.
  • domain assumption VideoMatte240K provides sufficient training signal for out-of-distribution alpha generation when combined with pretrained generative priors.
    The dataset contains 484 videos, mostly human subjects, yet examples include explosions, animals, and effects; Section 4 states the goal is to generate beyond the training set.
  • standard math Flow-matching or diffusion training objective is used for the fine-tuning.
    Section 3.2 says training uses flow matching or traditional diffusion; this is standard for these models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TransPixeler: Advancing Text-to-Video Generation with Transparency." pith.science (2026). https://pith.science/paper/J7EFOIZB

@misc{pith2026250103006,
  author       = {Pith},
  title        = {Pith review of: TransPixeler: Advancing Text-to-Video Generation with Transparency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7EFOIZB}},
  note         = {Machine review of arXiv:2501.03006}
}
read the original abstract

Text-to-video generative models have made significant strides, enabling diverse applications in entertainment, advertising, and education. However, generating RGBA video, which includes alpha channels for transparency, remains a challenge due to limited datasets and the difficulty of adapting existing models. Alpha channels are crucial for visual effects (VFX), allowing transparent elements like smoke and reflections to blend seamlessly into scenes. We introduce TransPixeler, a method to extend pretrained video models for RGBA generation while retaining the original RGB capabilities. TransPixar leverages a diffusion transformer (DiT) architecture, incorporating alpha-specific tokens and using LoRA-based fine-tuning to jointly generate RGB and alpha channels with high consistency. By optimizing attention mechanisms, TransPixar preserves the strengths of the original RGB model and achieves strong alignment between RGB and alpha channels despite limited training data. Our approach effectively generates diverse and consistent RGBA videos, advancing the possibilities for VFX and interactive content creation.

Figures

Figures reproduced from arXiv: 2501.03006 by the authors.

Figure 1
Figure 1. RGBA Video Generation with TransPixeler. By introducing LoRA layers into DiT-based text-to-video model with a novel alpha channel adaptive attention mechanism, our method enables RGBA video generation from text while preserving Text-to-Video quality. Abstract Text-to-video generative models have made significant strides, enabling diverse applications in entertainment, ad￾vertising, and education. However, generating… view at source ↗
Figure 2
Figure 2. Comparison between Generation-Then-Prediction and our Joint Generation approach. Given the generated RGB in (a), (b) and (c) show the predicted alpha (top) and the composited re￾sult (bottom). In (d), the top shows the jointly generated alpha. output structure and relying on the existing attention mech￾anism through LoRA adaptation. The extended sequence contains text, RGB, and alpha tokens, with self-attention divi… view at source ↗
Figure 3
Figure 3. Pipeline of TransPixeler. Our method is organized as follows: (1) Left: we extend the input of DiT to include new alpha tokens; (2) Top Center: we initialize alpha tokens with our positional encoding; (3) Bottom Center: we insert a partial LoRA and adjust attention computation during training and inference. are two commonly used types of positional encoding. One is absolute positional encoding formulated as follows:… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Positional Encoding Design for RGBA Generation. Assigning alpha tokens the same positional encoding as RGB yields similar results, resulting in faster convergence after 1000 iterations compared to standard encoding strategies. where γ controls the residual strength. Ad…
Figure 6
Figure 6. Figure 6: Applications. Top: Text-to-Video with Transparency. Bottom: Image-to-Video generation with transparency. . vide a diverse range of human subjects, clothing styles, and poses. We apply fundamental preprocessing steps for them, including color decontamination and backgro…
Figure 7
Figure 7. Figure 7: Comparison with Generation-then-Prediction Pipelines. Our method demonstrates superior alignment. “Swaying” “Flickering” “Running” “Turning Head” [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Comparison with Joint Generation Pipelines. Top: LayerDiffusion + AnimateDiff; Bottom: Ours. Our method achieves better alignment and generates corresponding motion described by prompts. mateDiff [17] with LayerDiffusion [65] to generate RGBA videos. We use the open-so…
Figure 11
Figure 11. Figure 11: Quantitative Evaluation. Our approach achieves a good balance between alignment (low flow difference) and pre￾serving generative quality (low FVD). but alternative extensions are possible at the Batch B or La￾tent Dimension D levels (see [PITH_FULL_IMAGE:figures/full…
Figure 10
Figure 10. Figure 10: Ablation Study. (a) Ours; (b) Ours without RGB￾attend-to-Alpha; (c) Ours with Text-attend-to-alpha; (d) Batch Ex￾tension Strategy; (e) Latent Dimension Extension Strategy. Our method maintains high-quality motion generation (e.g., butterflies waving their wings) while…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 27 canonical work pages

  1. [1]

    One transformer fits all distributions in multi-modal diffu- sion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffu- sion at scale. InInternational Conference on Machine Learn- ing, pages 1692–1717. PMLR, 2023. 3

    work page 2023

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

  3. [3]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  4. [4]

    Magick: A large-scale captioned dataset from matting generated images using chroma keying

    Ryan D Burgert, Brian L Price, Jason Kuen, Yijun Li, and Michael S Ryoo. Magick: A large-scale captioned dataset from matting generated images using chroma keying. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22595–22604, 2024. 3

    work page 2024

  5. [5]

    zeroscope v2

    cerspense. zeroscope v2. https://huggingface.co/ cerspense/zeroscope_v2_576w , 2023. Accessed: 2023-02-03. 3

    work page 2023

  6. [6]

    Pp-matting: high-accuracy natural image matting

    Guowei Chen, Yi Liu, Jian Wang, Juncai Peng, Yuying Hao, Lutao Chu, Shiyu Tang, Zewu Wu, Zeyu Chen, Zhiliang Yu, et al. Pp-matting: high-accuracy natural image matting. arXiv preprint arXiv:2204.09433, 2022. 3

    arXiv 2022

  7. [7]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 1, 3

    arXiv

  8. [8]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047, 2024. 3

    arXiv 2024

Show all 68 references
  1. [9]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 3

    arXiv

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 6

    2023 arXiv

  3. [11]

    Control-a-video: Controllable text-to-video generation with diffusion models

    Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023. 1

    2023 arXiv

  4. [12]

    Longnet: Scaling transformers to 1,000,000,000 tokens

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shao- han Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens

  5. [13]

    Two-frame motion estimation based on polynomial expansion

    Gunnar Farneb ¨ack. Two-frame motion estimation based on polynomial expansion. In Proceedings of the Scandina- vian Conference on Image Analysis (SCIA), pages 363–370. Springer, 2003. 8

    2003

  6. [14]

    Tokenflow: Consistent diffusion features for consistent video editing

    Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023. 1

    2023 arXiv

  7. [15]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 1

    2024 arXiv

  8. [16]

    I2v-adapter: A general image- to-video adapter for video diffusion models

    Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, et al. I2v-adapter: A general image- to-video adapter for video diffusion models. arXiv preprint arXiv:2312.16693, 2023. 1

    2023 arXiv

  9. [17]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 1, 3, 7

    2023 arXiv

  10. [18]

    Lu- cidfusion: Generating 3d gaussians with arbitrary unposed images, 2024

    Hao He, Yixun Liang, Luozhou Wang, Yuanhao Cai, Xinli Xu, Hao-Xiang Guo, Xiang Wen, and Yingcong Chen. Lu- cidfusion: Generating 3d gaussians with arbitrary unposed images, 2024. 3

    2024

  11. [19]

    Cameractrl: Enabling camera control for text-to-video generation

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024. 1

    2024 arXiv

  12. [20]

    Lotus: Diffusion-based visual foundation model for high-quality dense prediction

    Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Liu, Bingbing Liu, and Ying- Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409.18124, 2024. 2, 3, 6

    2024 arXiv

  13. [21]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022. 1

    2022 arXiv

  14. [22]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 5

    2020

  15. [23]

    Determining opti- cal flow

    Berthold KP Horn and Brian G Schunck. Determining opti- cal flow. Artificial intelligence, 17(1-3):185–203, 1981. 8

    1981

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 4

    2021 arXiv

  17. [25]

    Dreammotion: Space-time self-similarity score distillation for zero-shot video editing

    Hyeonho Jeong, Jinho Chang, Geon Yeong Park, and Jong Chul Ye. Dreammotion: Space-time self-similarity score distillation for zero-shot video editing. arXiv preprint arXiv:2403.12002, 2024. 1

    2024 arXiv

  18. [26]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9492...

    2024

  19. [27]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 1, 2, 3

    2024

  20. [28]

    Matting anything

    Jiachen Li, Jitesh Jain, and Humphrey Shi. Matting anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1775–1785, 2024. 3

    2024

  21. [29]

    Omnimat- terf: Robust omnimatte with 3d background modeling

    Geng Lin, Chen Gao, Jia-Bin Huang, Changil Kim, Yipeng Wang, Matthias Zwicker, and Ayush Saraf. Omnimat- terf: Robust omnimatte with 3d background modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23471–23480, 2023. 2, 3

    2023

  22. [30]

    Real-time high-resolution background matting

    Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher- Shlizerman. Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8762–8771, 2021. 2, 3...

    2021

  23. [31]

    Robust high-resolution video matting with tempo- ral guidance

    Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high-resolution video matting with tempo- ral guidance. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 238–247,

  24. [32]

    Motionclone: Training-free motion cloning for controllable video generation

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05338, 2024. 1

    2024 arXiv

  25. [33]

    Video-p2p: Video editing with cross-attention control

    Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8599–8608, 2024. 1

    2024

  26. [34]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 5

    2022 arXiv

  27. [35]

    Wonder3d: Sin- gle image to 3d using cross-domain diffusion

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Sin- gle image to 3d using cross-domain diffusion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  28. [36]

    Intrinsicdiffusion: Joint in- trinsic layers from latent diffusion models

    Jundan Luo, Duygu Ceylan, Jae Shin Yoon, Nanxuan Zhao, Julien Philip, Anna Fr ¨uhst¨uck, Wenbin Li, Christian Richardt, and Tuanfeng Wang. Intrinsicdiffusion: Joint in- trinsic layers from latent diffusion models. In ACM SIG- GRAPH 2024 Conference Papers, pages 1–11, 2024. 3

    2024

  29. [37]

    Trailblazer: Trajectory control for diffusion-based video generation

    Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. arXiv preprint arXiv:2401.00896, 2023. 1

    2023 arXiv

  30. [38]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 3

    2024 arXiv

  31. [39]

    Mofa-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model

    Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa-video: Control- lable image animation via generative motion field adaptions in frozen image-to-video diffusion model. arXiv preprint arXiv:2405.20222, 2024. 1

    2024 arXiv

  32. [40]

    Fatezero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15932–15942, 2023. 1

    2023

  33. [41]

    Bimatting: Efficient video matting via binarization

    Haotong Qin, Lei Ke, Xudong Ma, Martin Danelljan, Yu- Wing Tai, Chi-Keung Tang, Xianglong Liu, and Fisher Yu. Bimatting: Efficient video matting via binarization. Advances in Neural Information Processing Systems , 36: 43307–43321, 2023. 2, 6, 1

    2023

  34. [42]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  35. [43]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  36. [44]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  37. [45]

    Mochi, 2024

    Genmo Team. Mochi, 2024. 3, 4

    2024

  38. [46]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 8

    2019

  39. [47]

    Collaborative control for geometry-conditioned pbr image generation

    Shimon Vainer, Mark Boss, Mathias Parger, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Nicolas Per- ony, and Simon Donn ´e. Collaborative control for geometry-conditioned pbr image generation. arXiv preprint arXiv:2402.05919, 2024. 8

    2024 arXiv

  40. [48]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 1, 3

    2023 arXiv

  41. [49]

    Motion inversion for video customization

    Luozhou Wang, Guibao Shen, Yixun Liang, Xin Tao, Pengfei Wan, Di Zhang, Yijun Li, and Yingcong Chen. Motion inversion for video customization. arXiv preprint arXiv:2403.20193, 2024. 1

    2024 arXiv

  42. [50]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 1

    2006 arXiv

  43. [51]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  44. [52]

    Matting by gen- eration

    Zhixiang Wang, Baiang Li, Jian Wang, Yu-Lun Liu, Jinwei Gu, Yung-Yu Chuang, and Shin’Ichi Satoh. Matting by gen- eration. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 3

    2024

  45. [53]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Pa- pers, pages 1–11, 2024. 1

    2024

  46. [54]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  47. [55]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 3

    2024

  48. [56]

    Defect spectrum: A granular look of large-scale defect datasets with rich semantics, 2023

    Shuai Yang, Zhifei Chen, Pengguang Chen, Xi Fang, Shu Liu, and Yingcong Chen. Defect spectrum: A granular look of large-scale defect datasets with rich semantics, 2023. 3

    2023

  49. [57]

    Rerender a video: Zero-shot text-guided video-to-video translation

    Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers , pages 1–11, 2023. 1

    2023

  50. [58]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 1, 2, 3, 4, 6, 7

    2024 arXiv

  51. [59]

    Vitmatte: Boosting image matting with pre- trained plain vision transformers

    Jingfeng Yao, Xinggang Wang, Shusheng Yang, and Baoyuan Wang. Vitmatte: Boosting image matting with pre- trained plain vision transformers. Information Fusion, 103: 102091, 2024. 3

    2024

  52. [60]

    Matte anything: Interactive natural image matting with seg- ment anything model

    Jingfeng Yao, Xinggang Wang, Lang Ye, and Wenyu Liu. Matte anything: Interactive natural image matting with seg- ment anything model. Image and Vision Computing , 147: 105067, 2024. 6, 1

    2024

  53. [61]

    Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory

    Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023. 1

    2023 arXiv

  54. [62]

    Rgb ↔x: Image decomposition and synthesis using material-and lighting-aware diffusion models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloˇs Haˇsan. Rgb ↔x: Image decomposition and synthesis using material-and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 3

    2024

  55. [63]

    Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023. 1, 3

    2023 arXiv

  56. [64]

    Moonshot: To- wards controllable video generation and editing with multi- modal conditions

    David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sahoo. Moonshot: To- wards controllable video generation and editing with multi- modal conditions. arXiv preprint arXiv:2401.01827, 2024. 3

    2024 arXiv

  57. [65]

    Transparent image layer diffusion using latent transparency

    Lvmin Zhang and Maneesh Agrawala. Transparent image layer diffusion using latent transparency. arXiv preprint arXiv:2402.17113, 2024. 2, 3, 7

    2024 arXiv

  58. [66]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 1, 3

    2024

  59. [67]

    Long-short transformer: Efficient transformers for language and vision

    Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in neural information processing sys- tems, 34:17723–17736, 2021. 1

    2021

  60. [2023]

    arXiv preprint arXiv:2307.02486, 2023. 1

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.