Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Consistent Video Colorization via Palette Guidance

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that conditioning Stable Video Diffusion on a global five-color palette yields saturated, temporally consistent video colorization that beats prior automatic methods.

desk verdict Palette-guided SVD fine-tuning is a plausible new recipe for video colorization, but the evaluation undercuts the 'surpassing previous methods' claim. read the letter →

arxiv 2501.19331 v1 pith:VV4GUS5K submitted 2025-01-31 cs.CV

Han Wang , Yuang Zhang , Yuhong Zhang , Lingxiao Lu , Li Song This is my paper · ORCID
classification cs.CV
keywords videocolorizationdiffusionmodelpaletteguidancetemporalconsistencyStableDAVIS2017colorsaturationGaussianmixture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a video colorization pipeline that re-purposes Stable Video Diffusion (SVD) as a generative backbone and adds a global color palette as a conditioning signal. The palette, made of five dominant colors extracted from a reference frame, sampled from a Gaussian mixture model, or suggested by a language model, is turned into a color embedding that is broadcast into the first layer of the denoising U-Net. This embedding is intended to pull colors toward saturation and to keep the color style constant across frames and denoising windows. The authors claim that on the DAVIS 2017 benchmark the method surpasses prior automatic colorization methods in color saturation, image quality, and video quality, with the best scores on all metrics except PSNR.

What carries the argument

The central object is the palette-guider: a palette vector $C_{\text{palette}} \in \mathbb{R}^{1 \times 15}$ (five RGB colors) is projected through a linear layer into a color embedding $C_{\text{emb}} \in \mathbb{R}^{h \times w \times 320}$, which is spatially broadcast and added to the first-layer features of the denoising U-Net. This cheap conditioning mechanism carries the paper's argument: it injects a global color prior into every frame of the denoising process, decoupling color style from per-frame inference. The model fine-tunes the U-Net and the linear color network while freezing the VAE and CLIP encoder, and uses the progressive segment-based inference of MimicMotion to handle long videos.

What would settle it

A reader could rerun the DAVIS 2017 comparison with the diffusion-based colorization baselines cited in the related work ([18], [19], [20]) included, and compute Colorful and FVD over multiple random GMM palette draws; if the reported margins vanish or those baselines win, the claimed superiority is not robust.

Watch

Extended reading notes

Core claim

The central claim is that treating colorization as a generative task with SVD, conditioned by a global five-color palette, solves the two classic failure modes: unsaturated colors and temporal inconsistency. The paper's strongest stated result is in Section IV-B: 'Our method demonstrates significant advantages, achieving the best quantitative results in all metrics except PSNR.' The palette is generated by K-means clustering during training and, at inference, can come from a reference image, from random GMM sampling, or from an LLM that maps content tags to colors. The authors argue that this global guidance gives the model a unified color context across all frames, avoiding the drift and flicker caused by per-frame or per-clip color decisions.

Load-bearing premise

The benchmark evidence rests on one small dataset (DAVIS 2017's 90 training videos) with single point estimates and no significance tests, and the most relevant diffusion-based competitors from the related work are not included in the comparison table.

Editorial extensions

If this is right

  • If the claim holds, automatic old-film restoration can get saturated, long-range-consistent colors without optical-flow propagation, avoiding the cumulative error and color bleeding of prior methods.
  • The palette becomes a simple control knob: the same gray video can be re-colored in multiple plausible ways by swapping palettes, which suits the ill-posed nature of colorization.
  • A single global palette offers a cheap mechanism for temporal consistency that does not depend on neighboring-frame attention, potentially scaling to videos longer than the training clips.
  • An LLM-assisted palette connects semantic tags to colors, enabling content-aware colorization without paired exemplars or user-provided references.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the quantitative comparison excludes the diffusion-based baselines discussed in Related Work (notably [18], [19], [20]); including them could change the reported ranking, so the claimed superiority is open until such a comparison is run.
  • Beyond the paper: because the automatic GMM palette is sampled randomly, the colorization result is stochastic; reporting variance across palette draws would clarify whether the metric gaps (e.g., Colorful 22.64 vs 21.77) are stable or within run-to-run noise.
  • Beyond the paper: the global-palette mechanism is a generic conditioning trick that could anchor other video-to-video tasks needing a global style constant, such as relighting or day-to-night conversion, though the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an automatic video colorization method built by fine-tuning Stable Video Diffusion (SVD). A global color palette, obtained by K-means on a reference frame, is projected through a linear layer and added to the first U-Net encoder features to guide saturated and temporally consistent color generation. For long videos, the authors adopt a progressive segment-based generation with averaging of overlapping latents, following MimicMotion. The method is trained and evaluated on DAVIS 2017 (90 training videos) and compared against Deoldify, VCGAN, and TCVC. The authors report the best scores on Colorful, FID, FVD, SSIM, and LPIPS, and second-best on PSNR. An ablation study compares no-palette, GMM-sampled, GPT-generated, and GT-extracted palettes.

Significance. If the empirical claims hold, this is a useful contribution to video colorization: applying a strong video diffusion prior with a palette conditioning mechanism is a sensible and transferable idea, and the palette's ability to accept multiple input modalities (reference frame, GMM samples, LLM output) is practically valuable. The ablation in Table II gives some support for the necessity of palette guidance. However, the evaluation is too narrow to establish the paper's stated claim of surpassing previous methods: the most relevant diffusion-based baselines are absent from the quantitative comparison, and the reported point estimates lack uncertainty quantification. The work would be significant after a substantially strengthened evaluation.

major comments (4)
  1. [Section II-B and Section IV-B, Table I] The Related Work section identifies [18]-[20] as diffusion-based video colorization methods that address temporal consistency, yet none of these methods appears in the quantitative comparison in Table I. Since the abstract claims the method 'surpasses previous methods,' the omission of the closest prior work makes the central claim unsupported. The authors should add these baselines (or justify their exclusion with quantitative results) before claiming superiority.
  2. [Section IV-B, Table I] The reported metrics are point estimates with no error bars, confidence intervals, or significance tests. The margin over the nearest baseline is small for FID (53.76 vs. 54.28 for Deoldify) and LPIPS (0.1776 vs. 0.1798 for TCVC). Because the inference palette is randomly sampled from a GMM (Section III-C), the results may vary across runs; the authors should report mean and standard deviation over multiple sampled palettes and seeds, and perform a significance test.
  3. [Section III-A and Section III-C] The experimental setup is not reproducible as described. The progressive long-video generation paragraph does not state the segment length or the overlap used in the DAVIS experiments, and the GMM sampling procedure omits the number of components, covariance constraints, and the sampling seed. These choices can materially affect the quantitative results, so they should be specified.
  4. [Section IV] The evaluation is restricted to a single, small dataset (DAVIS 2017, 90 training videos) with no cross-dataset or old-film experiments. The paper motivates the task by old film restoration, but the test set does not reflect that domain. A single-domain, small-scale evaluation is insufficient to support the general claim of 'surpassing previous methods.'
minor comments (5)
  1. [Section IV-B] The word 'automantic' should be 'automatic.'
  2. [Figure 2 caption] The caption contains the garbled token 'Pale%e'; it should read 'Palette.'
  3. [Section III-B] In the definition Cpalette ∈ R^{1×15}, clarify that each of the five palette colors is an RGB triple, so the dimension is 5×3; using R^{5×3} would be clearer.
  4. [Section IV-C] In the sentence 'Experiments B − Dpreserve the original architecture', add a space: 'B − D preserve.'
  5. [References] Reference [33] (DeOldify) is a GitHub project without a version or year; provide a dated citation or access date for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claim is an external benchmark comparison; the only self-citation (MimicMotion [50]) is used for a progressive inference strategy and is not load-bearing.

full rationale

The paper's derivation chain is not circular. The method fine-tunes Stable Video Diffusion with a palette-guidance branch (Eqs. 1-5); the colorization output is evaluated on DAVIS 2017 against Deoldify, VCGAN, and TCVC (Table I). The palette is either extracted via K-means from a reference frame, sampled from a GMM fit to training pixels, or generated by GPT, and the quantitative results do not reduce to any fitted parameter or self-defined quantity. The single self-citation, 'Inspired by MimicMotion [50], we adopt the progressive approach for generating long videos' (Section III-A), is a borrowed inference strategy rather than evidence for the central claim; even if MimicMotion were absent, the described segment-and-average procedure stands on its own. The Section V limitation about lacking instance-level color control is an honest scope statement, not a circularity. One robustness concern, which is a correctness risk rather than circularity, is that Section IV-B omits the diffusion-based video colorization baselines [18]-[20] cited in Related Work, so the blanket claim of 'surpassing previous methods' is only supported against non-diffusion baselines; this does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The palette guide is a conditioning signal, not a new physical or conceptual entity. The paper introduces no new particles, forces, dimensions, or conserved quantities; the palette embedding is a learned network component, not an independent postulated entity.

free parameters (4)
  • Palette size = 5
    The palette is fixed to five colors (Section III-B), defining the dimension of the global color embedding and controlling the diversity and strength of color guidance.
  • Saturation filter variance threshold = 50
    Pixels with RGB value variance below 50 are removed before GMM fitting (Section III-C), which shapes the distribution from which automatic palettes are sampled.
  • GMM component count = not reported
    The Gaussian mixture model used for automatic palette sampling is fitted to training pixels, but the number of components is not given; this setting controls the variety of sampled palettes.
  • Progressive segment length and overlap = not reported
    For long videos, the sequence is split into fixed-length segments with overlapped latent averaging (Section III-A), but no numbers are provided; these choices directly affect temporal consistency.
assumptions (5)
  • domain assumption Stable Video Diffusion 1.1 provides a strong video prior that can be fine-tuned for colorization.
    The whole pipeline relies on the pretrained SVD weights as the starting point (Section III-A); no evidence in this paper independently establishes that this prior transfers to colorization.
  • domain assumption K-means clustering of a reference frame yields a representative five-color palette.
    Used to create training palettes (Section III-B) and inference palette method (a); the paper does not analyze how palette quality varies with clustering settings.
  • domain assumption A GMM fitted to filtered training pixels produces plausible palettes for unseen videos.
    Automatic colorization in the quantitative experiments samples palettes from this GMM (Section III-C); if the DAVIS training distribution is unrepresentative, the automatic mode fails.
  • ad hoc to paper Adding a linearly projected palette embedding to the first U-Net layer is sufficient to guide global colorization.
    Equation (4) is an architectural choice without derivation; the paper relies on the ablation to justify it.
  • domain assumption DAVIS 2017 is representative of general video colorization tasks.
    Training (90 videos) and evaluation are both on DAVIS 2017 (Section IV-A/B); the paper makes general claims beyond this benchmark without cross-dataset validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consistent Video Colorization via Palette Guidance." pith.science (2026). https://pith.science/paper/VV4GUS5K

@misc{pith2026250119331,
  author       = {Pith},
  title        = {Pith review of: Consistent Video Colorization via Palette Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VV4GUS5K}},
  note         = {Machine review of arXiv:2501.19331}
}
read the original abstract

Colorization is a traditional computer vision task and it plays an important role in many time-consuming tasks, such as old film restoration. Existing methods suffer from unsaturated color and temporally inconsistency. In this paper, we propose a novel pipeline to overcome the challenges. We regard the colorization task as a generative task and introduce Stable Video Diffusion (SVD) as our base model. We design a palette-based color guider to assist the model in generating vivid and consistent colors. The color context introduced by the palette not only provides guidance for color generation, but also enhances the stability of the generated colors through a unified color context across multiple sequences. Experiments demonstrate that the proposed method can provide vivid and stable colors for videos, surpassing previous methods.

Figures

Figures reproduced from arXiv: 2501.19331 by the authors.

Figure 1
Figure 1. Comparison with frame independent colorization framework TCVC [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the video colorization pipeline with palette-guidance. We re-purpose a pre-trained video diffusion model and augment it with palette [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visual comparison of colorization results for 2 videos. From left to right, the colorization results for Deoldify [33], VCGAN [8], TCVC [9], and ours. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visual effects of ablation studies. The first column (NULL) shows the colorization results without palette guidance. The other columns display the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Diverse colorization results with distinct palettes. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 44 canonical work pages

  1. [18]

    Video Colorization with Pre-trained Text-to-Image Diffusion Models

    Hanyuan Liu, Minshan Xie, Jinbo Xing, Chengze Li, and Tien-Tsin Wong, “Video colorization with pre-trained text-to-image diffusion models,” arXiv preprint arXiv:2306.01732 , 2023

    work page Pith review arXiv 2023

  2. [20]

    Versatile vision foundation model for image and video colorization,

    Vukasin Bozic, Abdelaziz Djelouah, Yang Zhang, Radu Timofte, Markus Gross, and Christopher Schroers, “Versatile vision foundation model for image and video colorization,” in ACM SIGGRAPH 2024 Conference Papers, 2024, pp. 1–11

    work page 2024

  3. [1]

    Chromagan: Adversarial picture colorization with semantic class distribution,

    Patricia Vitoria, Lara Raad, and Coloma Ballester, “Chromagan: Adversarial picture colorization with semantic class distribution,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 2445–2454

    work page 2020

  4. [2]

    Ct2: Colorization transformer via color tokens,

    Shuchen Weng, Jimeng Sun, Yu Li, Si Li, and Boxin Shi, “Ct2: Colorization transformer via color tokens,” in ECCV, 2022

    work page 2022

  5. [3]

    Colorization transformer,

    Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner, “Colorization transformer,” in International Conference on Learning Representations , 2021

    work page 2021

  6. [4]

    Bigcolor: colorization using a generative color prior for natural images,

    Geonung Kim, Kyoungkook Kang, Seongtae Kim, Hwayoon Lee, Se- hoon Kim, Jonghyun Kim, Seung-Hwan Baek, and Sunghyun Cho, “Bigcolor: colorization using a generative color prior for natural images,” in European Conference on Computer Vision . Springer, 2022, pp. 350– 366

    work page 2022

  7. [5]

    Unicolor: A unified framework for multi-modal colorization with transformer,

    Zhitong Huang, Nanxuan Zhao, and Jing Liao, “Unicolor: A unified framework for multi-modal colorization with transformer,” ACM Trans- actions on Graphics (TOG) , vol. 41, no. 6, pp. 1–16, 2022

    work page 2022

  8. [6]

    L-cad: Language-based colorization with any-level descriptions using diffusion priors,

    Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, Boxin Shi, et al., “L-cad: Language-based colorization with any-level descriptions using diffusion priors,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

Show all 58 references
  1. [7]

    Fully automatic video colorization with self-regularization and diversity,

    Chenyang Lei and Qifeng Chen, “Fully automatic video colorization with self-regularization and diversity,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 3753– 3761

    2019

  2. [8]

    Vcgan: Video coloriza- tion with hybrid generative adversarial network,

    Yuzhi Zhao, Lai-Man Po, Wing-Yin Yu, Yasar Abbas Ur Rehman, Mengyang Liu, Yujia Zhang, and Weifeng Ou, “Vcgan: Video coloriza- tion with hybrid generative adversarial network,” IEEE Transactions on Multimedia, vol. 25, pp. 3017–3032, 2023

    2023

  3. [9]

    Temporally consistent video colorization with deep feature propagation and self-regularization learning,

    Yihao Liu, Hengyuan Zhao, Kelvin CK Chan, Xintao Wang, Chen Change Loy, Yu Qiao, and Chao Dong, “Temporally consistent video colorization with deep feature propagation and self-regularization learning,” Computational Visual Media , vol. 10, no. 2, pp. 375–395, 2024

    2024

  4. [10]

    Flownet 2.0: Evolution of optical flow estimation with deep networks,

    Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2462–2470

    2017

  5. [11]

    Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,

    Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz, “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8934–8943

    2018

  6. [12]

    Zero-shot image restoration using denoising diffusion null-space model,

    Yinhuai Wang, Jiwen Yu, and Jian Zhang, “Zero-shot image restoration using denoising diffusion null-space model,” The Eleventh International Conference on Learning Representations , 2023

    2023

  7. [13]

    Instructpix2pix: Learning to follow image editing instructions,

    Tim Brooks, Aleksander Holynski, and Alexei A Efros, “Instructpix2pix: Learning to follow image editing instructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 18392–18402

    2023

  8. [14]

    Diffir: Efficient diffusion model for image restoration,

    Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool, “Diffir: Efficient diffusion model for image restoration,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 13095–13105

    2023

  9. [15]

    Control color: Multimodal diffusion-based inter- active image colorization,

    Zhexin Liang, Zhaochen Li, Shangchen Zhou, Chongyi Li, and Chen Change Loy, “Control color: Multimodal diffusion-based inter- active image colorization,” arXiv preprint arXiv:2402.10855 , 2024

    2024 arXiv

  10. [16]

    Auto- matic controllable colorization via imagination,

    Xiaoyan Cong, Yue Wu, Qifeng Chen, and Chenyang Lei, “Auto- matic controllable colorization via imagination,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 2609–2619

    2024

  11. [17]

    Multicolor: Image colorization by learning from multiple color spaces,

    Xiangcheng Du, Zhao Zhou, Xingjiao Wu, Yanlong Wang, Zhuoyao Wang, Yingbin Zheng, and Cheng Jin, “Multicolor: Image colorization by learning from multiple color spaces,” in Proceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 6784–6792

    2024

  12. [19]

    Towards photorealistic video colorization via gated color-guided image diffusion models,

    Jiaxing Li, Hongbo Zhao, Yijun Wang, and Jianxin Lin, “Towards photorealistic video colorization via gated color-guided image diffusion models,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 10891–10900

    2024

  13. [21]

    Lvcd: reference-based lineart video colorization with diffusion models,

    Zhitong Huang, Mohan Zhang, and Jing Liao, “Lvcd: reference-based lineart video colorization with diffusion models,” ACM Transactions on Graphics (TOG), vol. 43, no. 6, pp. 1–11, 2024

    2024

  14. [22]

    Gaussian mixture models.,

    Douglas A Reynolds et al., “Gaussian mixture models.,” Encyclopedia of biometrics , vol. 741, no. 659-663, 2009

    2009

  15. [23]

    Gpt-4 technical report,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  16. [24]

    Focusing on persons: Colorizing old images learning from modern historical movies,

    Xin Jin, Zhonglan Li, Ke Liu, Dongqing Zou, Xiaodong Li, Xingfan Zhu, Ziyin Zhou, Qilong Sun, and Qingyu Liu, “Focusing on persons: Colorizing old images learning from modern historical movies,” in Proceedings of the 29th ACM International Conference on Multimedia , 2021, pp. ...

    2021

  17. [25]

    Ddcolor: Towards photo-realistic image colorization via dual decoders,

    Xiaoyang Kang, Tao Yang, Wenqi Ouyang, Peiran Ren, Lingzhi Li, and Xuansong Xie, “Ddcolor: Towards photo-realistic image colorization via dual decoders,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 328–338

    2023

  18. [26]

    High-resolution image synthesis with latent diffu- sion models,

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer, “High-resolution image synthesis with latent diffu- sion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10684–10695

    2022

  19. [27]

    Deep exemplar-based video colorization,

    Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen, “Deep exemplar-based video colorization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 8052–8061

    2019

  20. [28]

    Deepremaster: temporal source- reference attention networks for comprehensive video enhancement,

    Satoshi Iizuka and Edgar Simo-Serra, “Deepremaster: temporal source- reference attention networks for comprehensive video enhancement,” ACM Transactions on Graphics (TOG) , vol. 38, no. 6, pp. 1–13, 2019

    2019

  21. [29]

    Swintexco: Exemplar-based video colorization using swin transformer,

    Duong Thanh Tran, Nguyen Doan Hieu Nguyen, Trung Thanh Pham, Phuong-Nam Tran, Thuy-Duong Thi Vu, Cuong Tuan Nguyen, Hanh Dang-Ngoc, and Duc Ngoc Minh Dang, “Swintexco: Exemplar-based video colorization using swin transformer,” Expert Systems with Appli- cations, vol. 260, pp. ...

    2025

  22. [30]

    Exemplar-based video colorization with long-term spatiotemporal dependency,

    Siqi Chen, Xueming Li, Xianlin Zhang, Mingdao Wang, Yu Zhang, Jiatong Han, and Yue Zhang, “Exemplar-based video colorization with long-term spatiotemporal dependency,” Knowledge-Based Systems , vol. 284, pp. 111240, 2024

    2024

  23. [31]

    Colormnet: A memory-based deep spatial-temporal feature propagation network for video colorization,

    Yixin Yang, Jiangxin Dong, Jinhui Tang, and Jinshan Pan, “Colormnet: A memory-based deep spatial-temporal feature propagation network for video colorization,” in European Conference on Computer Vision . Springer, 2025, pp. 336–352

    2025

  24. [32]

    Bistnet: Semantic image prior guided bidirectional temporal feature fusion for deep exemplar-based video colorization,

    Yixin Yang, Jinshan Pan, Zhongzheng Peng, Xiaoyu Du, Zhulin Tao, and Jinhui Tang, “Bistnet: Semantic image prior guided bidirectional temporal feature fusion for deep exemplar-based video colorization,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  25. [33]

    DeOldify: A Deep Learning based project for colorizing and restoring old images (and video!),

    Jason Antic, “DeOldify: A Deep Learning based project for colorizing and restoring old images (and video!),”

  26. [34]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning,

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai, “Animatediff: Animate your personalized text- to-image diffusion models without specific tuning,” arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  27. [35]

    Sparsectrl: Adding sparse controls to text-to-video diffu- sion models,

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai, “Sparsectrl: Adding sparse controls to text-to-video diffu- sion models,” in European Conference on Computer Vision . Springer, 2025, pp. 330–348

    2025

  28. [36]

    Align your latents: High-resolution video synthesis with latent diffusion models,

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Se- ung Wook Kim, Sanja Fidler, and Karsten Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023

    2023

  29. [37]

    Videocrafter1: Open diffusion models for high-quality video generation,

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al., “Videocrafter1: Open diffusion models for high-quality video generation,” arXiv preprint arXiv:2310.19512 , 2023

    arXiv

  30. [38]

    Stable video diffusion: Scaling latent video diffusion models to large datasets,

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al., “Stable video diffusion: Scaling latent video diffusion models to large datasets,” arXiv preprint arXiv:2311.15127 , 2023

    2023 arXiv

  31. [39]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models,

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan, “Videocrafter2: Overcoming data limitations for high-quality video diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 7310–7320

    2024

  32. [40]

    Cogvideox: Text-to-video diffusion models with an expert transformer,

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al., “Cogvideox: Text-to-video diffusion models with an expert transformer,” arXiv preprint arXiv:2408.06072 , 2024

    2024 arXiv

  33. [41]

    Hunyuanvideo: A systematic framework for large video generative models,

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al., “Hunyuanvideo: A systematic framework for large video generative models,” arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  34. [42]

    Lora: Low-rank adaptation of large language models,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  35. [43]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation,

    Li Hu, “Animate anyone: Consistent and controllable image-to-video synthesis for character animation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 8153–8163

    2024

  36. [44]

    Dreamvideo: Composing your dream videos with customized subject and motion,

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan, “Dreamvideo: Composing your dream videos with customized subject and motion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  37. [45]

    Champ: Controllable and consistent human image animation with 3d parametric guidance,

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu, “Champ: Controllable and consistent human image animation with 3d parametric guidance,” in European Conference on Computer Vision. Springer, 2025, pp. 145–162

    2025

  38. [46]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions,

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo, “Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions,” in European Conference on Computer Vision. Springer, 2025, pp. 244–260

    2025

  39. [47]

    Motionctrl: A unified and flexible motion controller for video generation,

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan, “Motionctrl: A unified and flexible motion controller for video generation,” in ACM SIGGRAPH 2024 Conference Papers , 2024, pp. 1–11

    2024

  40. [48]

    Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling,

    Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al., “Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling,” in ACM SIGGRAPH 2024 Conference Papers,...

    2024

  41. [49]

    Motiondirector: Motion customization of text-to-video diffusion models,

    Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou, “Motiondirector: Motion customization of text-to-video diffusion models,” in European Conference on Computer Vision . Springer, 2025, pp. 273–290

    2025

  42. [50]

    Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance,

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou, “Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance,” arXiv preprint arXiv:2406.19680, 2024

    2024 arXiv

  43. [51]

    Dynamicrafter: Animating open-domain images with video diffusion priors,

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong, “Dynamicrafter: Animating open-domain images with video diffusion priors,” in European Conference on Computer Vision. Springer, 2025, pp. 399–417

    2025

  44. [52]

    Learning transferable visual models from natural language supervision,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning ...

    2021

  45. [53]

    Recognize anything: A strong image tagging model,

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al., “Recognize anything: A strong image tagging model,” arXiv preprint arXiv:2306.03514, 2023

    2023 arXiv

  46. [54]

    The 2017 davis challenge on video object segmentation,

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbel ´aez, Alex Sorkine-Hornung, and Luc Van Gool, “The 2017 davis challenge on video object segmentation,” arXiv preprint arXiv:1704.00675 , 2017

    2017 arXiv

  47. [55]

    Measuring colorfulness in natural images,

    David Hasler and Sabine E Suesstrunk, “Measuring colorfulness in natural images,” in Human vision and electronic imaging VIII . SPIE, 2003, vol. 5007, pp. 87–95

    2003

  48. [56]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems , vol. 30, 2017

    2017

  49. [57]

    Fvd: A new metric for video generation,

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha ¨el Marinier, Marcin Michalski, and Sylvain Gelly, “Fvd: A new metric for video generation,” in DGS@ICLR, 2019

    2019

  50. [58]

    The unreasonable effectiveness of deep features as a perceptual metric,

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595

    2018

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.