Pith. sign in

REVIEW 6 major objections 5 minor 26 references

SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation

T0 review · 6 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper claims that a diffusion transformer with channel-concatenated sketch latents produces the best sketch-to-colour results on the SAKUGA dataset while training on roughly half the data of earlier models.

desk verdict A sensible DiT-based sketch colorization idea with a clean architecture story, but the evaluation protocol and reported statistics do not support the abstract's across-all-metrics claim. read the letter →

arxiv 2507.01586 v1 pith:V3UBBF54 submitted 2025-07-02 cs.CV

classification cs.CV
keywords sketchcolorizationdiffusiontransformer2DanimationvideoLoRAfine-tuningchannelconcatenationControlNettemporalcoherence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SketchColour sets out to show that a diffusion transformer (DiT) can replace the usual U-Net-plus-ControlNet stack for sketch-to-colour animation. The paper claims this is the first DiT-based pipeline for the task, and that on the SAKUGA dataset it beats prior video colourisation methods on every reported metric while training on roughly half their data. The practical goal is that an animator colors only the first keyframe and the model colors the remaining sketches with temporal coherence and minimal colour bleed.

What carries the argument

The load-bearing object is the channel-concatenation adapter on a DiT backbone. The colored first frame, the noisy video latent, and the sketch sequence are each encoded by the same frozen 3D VAE, then concatenated along the channel dimension; the projection for the new sketch channel is zero-initialized in the style of ControlNet, while LoRA updates the attention projections and feedforward layers. This replaces the duplicated ControlNet that earlier U-Net pipelines required, and it is what the paper credits for lower parameter count, lower GPU memory, and fewer colour-bleed artifacts.

What would settle it

Decode the frozen VAE latent of a sketch back to pixels and compare it with the original line art: if thin strokes or small features vanish, the conditioning channel is already corrupted, and an ablation with a separately trained sketch encoder would reveal what the pipeline is losing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a frozen 3D VAE encodes line-art sketches faithfully enough to serve as a conditioning signal, so the sketch latent can be concatenated channel-wise with the colored reference latent and the noisy video latent as input to a pretrained DiT video model. A zero-initialized patch projection adds the new sketch channel, and a small LoRA (about 10 million parameters) fine-tunes attention. With this setup, SketchColour reports the best MSCE, PSNR, SSIM, LPIPS, and FVD results at matched frame counts against AniDoc, LVCD, and ToonCrafter on SAKUGA, and attributes the gain to the DiT's global attention scope, which avoids the latent-gap colour bleed seen in ControlNet baselines.

Load-bearing premise

The load-bearing premise is that the frozen 3D VAE encodes sketch line art into latents that still carry enough detail, so channel-wise concatenation can supply the colouring condition without a dedicated sketch encoder.

Editorial extensions

If this is right

  • With one colored keyframe and a sketch sequence as input, the pipeline produces a fully colored, temporally coherent animation clip.
  • Because conditioning is injected by channel concatenation plus a roughly 10-million-parameter LoRA, the method avoids the billions of parameters a duplicated ControlNet adds.
  • The reported SAKUGA results indicate the DiT backbone can outperform U-Net/ControlNet baselines across all five metrics while training on about 80K video clips.
  • The frozen 3D VAE is claimed to need no dedicated sketch encoder, so a full sketch-conditioned model can be obtained by fine-tuning a small adapter set.
  • Global attention in the DiT reduces colour bleed and object deformation compared with the spatial down-sampling U-Net baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same channel-concat recipe could carry other sparse conditioning modalities, such as depth, pose, or motion hints, into DiT video generators without a ControlNet.
  • If the frozen-VAE claim holds, the cost of adding a new sparse control signal to a new DiT model drops to a small adapter and LoRA training run.
  • A natural extension the paper does not test is sliding-window propagation of the colored frame to much longer clips, which would test whether the temporal coherence holds beyond 17 frames.
  • The method does not address deliberate palette changes mid-scene; combining it with text descriptions attached to the dataset could give animators explicit color control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. SketchColour proposes a DiT-based pipeline for sketch-to-colour animation generation. The method replaces the U-Net denoiser of prior sketch-colorization systems with CogVideoX-I2V, a diffusion transformer, and injects sketch conditioning by channel-wise concatenation of the frozen VAE latents of the sketch sequence with the reference-frame and noisy-video latents, plus LoRA fine-tuning of attention and feed-forward layers. Evaluated on a subsample of the SAKUGA dataset, the paper reports that SketchColour outperforms AniDoc, LVCD, and ToonCrafter on MSCE, PSNR, SSIM, LPIPS, and FVD at frame lengths matching the baselines, while using fewer trainable parameters (10M LoRA) and, per the abstract, only half the training data.

Significance. If the reported results are reproducible, the paper makes a useful contribution to animation production: it is the first sketch-to-colour pipeline on a DiT backbone, it avoids a duplicated ControlNet, and the quantitative gains on colorization fidelity metrics are meaningful. The method is parameter-efficient and could lower the barrier for practical adoption. However, the contribution is primarily empirical, and several load-bearing claims—the across-all-metrics superiority, the data-efficiency comparison, and the frozen-VAE fidelity—are not sufficiently verified by the manuscript as written. The paper also lacks evaluation artifacts (seed, clip list, code) needed to confirm the numbers.

major comments (6)
  1. [Abstract; §4.3, Table 1] The abstract claims SketchColour 'outperforms previous state-of-the-art video colourization methods across all metrics,' but Table 1 and §4.3 show that at the model's native 17-frame length, SketchColour's FVD is 918.70±771.13, which is worse than AniDoc's 14-frame FVD of 898.19±704.30; the text itself states that on FVD the model 'only loses to AniDoc with a slight margin.' Because the 17-frame row is the native output of the model and the comparison against AniDoc is at a different frame length, the across-all-metrics claim is contradicted by the paper's own results and must be qualified or removed.
  2. [§4.1, Table 1] The evaluation protocol is not reproducible as described. The paper uses a random 1K-clip subsample of a filtered SAKUGA test set, but it does not provide the sampling seed, the list of clips, or the evaluation code; the provided link is a project page, not a code repository. It is also not stated whether the AniDoc, LVCD, and ToonCrafter numbers in Table 1 were produced by the authors on this same 1K split using official checkpoints and hyperparameters, or whether they were transcribed from prior papers. Without this information, the quantitative comparison cannot be independently verified.
  3. [Abstract; §4.1] The abstract's claim that SketchColour uses 'only half the training data of competing models' is unsupported. Section 4.1 reports that SketchColour was trained on 80K sampled SAKUGA videos, but the training dataset sizes for AniDoc, LVCD, and ToonCrafter are never given. The relative data-efficiency claim should be substantiated with concrete numbers for the baselines or removed.
  4. [§4.2, Figure 4] The claim that the frozen 3D VAE preserves sketch information is supported only by a PCA visualization, which shows qualitative resemblance but provides no quantitative measure of reconstruction fidelity or of the specific line-art details needed for colorization. This assumption is load-bearing because channel-concat conditioning depends on the sketch latent retaining fine structure. The paper should provide quantitative evidence of sketch-latent fidelity (e.g., reconstruction error or downstream metric comparison) or an ablation, such as training with a dedicated sketch encoder, to justify the assertion that 'there is no need to fine-tune a specialized sketch encoder.'
  5. [§4.1, Table 1] The table reports SketchColour results at 14, 16, and 17 frames, while the model is trained to generate clips of 17 frames. The paper does not explain how the 14- and 16-frame outputs are produced (truncation, padding, re-generation, or a separate inference protocol), nor does it state whether the baselines were evaluated on the same 1K clips at native resolution with identical preprocessing. This ambiguity undermines the 'fair comparison' statement in the table caption.
  6. [§3.1, §4.1] The parameter-efficiency claim—'LoRA of only 10 million parameters compared to the billions of parameters used by ControlNet'—is incomplete. The method also trains new zero-initialized patch-projection weights for the added sketch channel; whether these weights are included in the 10M count is not specified. No direct comparison of total trainable parameters, GPU memory usage, or training time against the baselines is reported, so the stated contribution of 'reducing parameter count and GPU memory usage' is not fully quantified.
minor comments (5)
  1. [Abstract] 'The production of high-quality 2D animation is highly labor-intensive process' should read 'a highly labor-intensive process.'
  2. [§4.1] 'we used a Lora of rank 192' should be 'LoRA,' and 'DDP training' should be spelled out as distributed data-parallel training.
  3. [Figure 4 caption] 'repsectively' is a typo for 'respectively'; the abbreviation 'VAE' is inconsistently spaced as 'V AE' throughout the paper.
  4. [§4.3] The sentence 'with the score difference against the baselines being half of more of those baselines' standard deviation' is unclear and should be rewritten to specify which differences are being compared.
  5. [References, §1] Reference [18] has a malformed author list, and [25] appears to list author names in the wrong order; the project page link in the abstract is described as 'code' but does not appear to link to an implementation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims rest on external benchmarks and architectural comparisons, not on self-referential derivations.

full rationale

SketchColour is an empirical systems paper. The proposed method concatenates frozen 3D VAE latents of the colored reference frame and sketch frames, fine-tunes a DiT backbone with LoRA, and evaluates on external SAKUGA test clips using standard metrics. None of the core claims reduce to the paper's inputs by construction: the quantitative results in Table 1 come from held-out test videos and public baseline models, not from fitted parameters that are then renamed as predictions. The claim that a frozen VAE can encode sketch line art is supported by a PCA visualization rather than a rigorous proof, but this is an evidence-strength limitation, not a circular step. The paper also contains an internal inconsistency: the abstract claims improvement 'across all metrics,' while Section 4.3 admits that on FVD at 17 frames the method 'only loses to AniDoc with a slight margin.' That is an overclaim or a reporting error, not a circularity. The comparison protocol is under-specified (custom 1K sample, no sampling seed, unspecified baseline reproduction conditions), which threatens verifiability but does not make the derivation circular. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation of the authors' own prior work. Accordingly, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method relies on domain assumptions about the frozen VAE's ability to encode sketches, the suitability of the pretrained I2V prior, the fidelity of the sketch extraction, and the fairness of the evaluation split. No new physical or mathematical entities are introduced. Training hyperparameters are chosen by hand and could affect the reported gains.

free parameters (5)
  • LoRA rank = 192
    Chosen by hand to trade adapter capacity against trainable parameter count; no ablation is reported.
  • Learning rate = 1e-4
    AdamW learning rate used for fine-tuning; not justified by ablations.
  • Training steps = 40,000
    Number of DDP update steps; chosen based on available compute, no convergence analysis.
  • Batch size = 2
    Per-GPU batch size; fixed by memory constraints.
  • Frame length = 17
    Fixed to the minimum CogVideoX generation length; test at 14 and 16 frames for comparison.
assumptions (5)
  • domain assumption The frozen 3D VAE preserves sketch line-art structure well enough for channel-concat conditioning.
    Section 4.2 claims the sketch latent resembles the sketched RGB latent, but only a PCA visualization is shown, no quantitative reconstruction or downstream ablation.
  • domain assumption CogVideoX-I2V-5B provides a suitable pretrained prior for 2D animation colorization.
    The paper relies on pretrained I2V knowledge without assessing domain shift to anime line art.
  • domain assumption Anime2Sketch plus binarization produces sketches comparable to those used by prior baselines.
    Section 3.2 adopts Anime2Sketch for fair comparison; no analysis of sketch distribution alignment with baseline inputs.
  • domain assumption The sampled 80K/1K split of SAKUGA is a fair basis for comparing against baselines.
    Section 4.1 describes random sampling after filtering; the text does not show the baselines were re-run on exactly the same clips.
  • domain assumption Standard metrics (MSCE, PSNR, SSIM, LPIPS, FVD) capture the practical quality of animation colorization.
    These are generic reconstruction/video metrics; no user study or animator evaluation is included.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation." pith.science (2026). https://pith.science/paper/V3UBBF54

@misc{pith2026250701586,
  author       = {Pith},
  title        = {Pith review of: SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3UBBF54}},
  note         = {Machine review of arXiv:2507.01586}
}
read the original abstract

The production of high-quality 2D animation is highly labor-intensive process, as animators are currently required to draw and color a large number of frames by hand. We present SketchColour, the first sketch-to-colour pipeline for 2D animation built on a diffusion transformer (DiT) backbone. By replacing the conventional U-Net denoiser with a DiT-style architecture and injecting sketch information via lightweight channel-concatenation adapters accompanied with LoRA finetuning, our method natively integrates conditioning without the parameter and memory bloat of a duplicated ControlNet, greatly reducing parameter count and GPU memory usage. Evaluated on the SAKUGA dataset, SketchColour outperforms previous state-of-the-art video colourization methods across all metrics, despite using only half the training data of competing models. Our approach produces temporally coherent animations with minimal artifacts such as colour bleeding or object deformation. Our code is available at: https://bconstantine.github.io/SketchColour .

Figures

Figures reproduced from arXiv: 2507.01586 by the authors.

Figure 1
Figure 1. SketchColour receives the colored first frame and the entire scene in sketch format, then colors each frame based on the reference. Abstract The production of high-quality 2D animation is highly labor-intensive process, as animators are currently required to draw and color a large number of frames by hand. We present SketchColour, the first sketch-to-colour pipeline for 2D animation built on a diffusion transformer … view at source ↗
Figure 2
Figure 2. Due to the frame-by-frame workflow of 2D an [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The SketchColour model pipeline. Our model uses a frozen VAE to encode both the colored first frame and the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: A representation of a video scene on the top row [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Visual comparison of sketch colorization work with colored first frame as a reference, compared to ToonCrafter [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 17 canonical work pages

  1. [1]

    https://zhenglinpan.github.io/ AnitaDataset_homepage/

    Anita dataset. https://zhenglinpan.github.io/ AnitaDataset_homepage/. Accessed: 2025-06-13. 1

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. 2, 3

  3. [3]

    Yu Cao, Xiangqiao Meng, P. Y . Mok, Tong-Yee Lee, Xuet- ing Liu, and Ping Li. Animediffusion: Anime diffusion col- orization. IEEE Transactions on Visualization and Computer Graphics, 30(10):6956–6969, Oct. 2024. 2

  4. [4]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. 2022. 2

  5. [5]

    Lvcd: Reference-based lineart video colorization with diffusion models

    Zhitong Huang, Mohan Zhang, and Jing Liao. Lvcd: Reference-based lineart video colorization with diffusion models. ACM Transactions on Graphics, 43(6):1–11, Nov

  6. [6]

    Tag2pix: Line art colorization using text tag with secat and changing loss

    Hyunsu Kim, Ho Young Jhoo, Eunhyeok Park, and Sungjoo Yoo. Tag2pix: Line art colorization using text tag with secat and changing loss. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9056–9065, Seoul, South Korea, October 2019. 3

  7. [7]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. CoRR, abs/2412.03603, 2024. 2

  8. [8]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 4

Show all 26 references
  1. [9]

    Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance

    Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu. Magicmotion: Controllable video generation with dense-to-sparse trajectory guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Honolulu, Hawai‘i, USA, October 2025. 1

  2. [10]

    Eliminating gradient conflict in reference-based line-art colorization

    Zekun Li, Zhengyang Geng, Zhao Kang, Wenyu Chen, and Yibo Yang. Eliminating gradient conflict in reference-based line-art colorization. In European Conference on Computer Vision, pages 579–596. Springer, 2022. 2

  3. [11]

    Motionclone: Training-free motion cloning for controllable video generation

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. In The Thirteenth International Confer- ence on Learning Representations, 2025. 1

  4. [12]

    Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, Lifang He, and Lichao Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024. 2

  5. [13]

    Anidoc: Animation creation made easier

    Yihao Meng, Hao Ouyang, Hanlin Wang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Zhiheng Liu, Yujun Shen, and Huamin Qu. Anidoc: Animation creation made easier. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference (CVPR), pages 18187–18197, June 2025. 1, 2, 3, 4, 6

  6. [14]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 4195–4205, October 2023. 2

  7. [15]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 5

  8. [16]

    Wan: Open and advanced large-scale video generative models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, P...

  9. [17]

    Framer: Interactive video interpolation

    Wen Wang, Qiuyu Wang, Kecheng Zheng, Hao Ouyang, Zhekai Chen, Biao Gong, Hao Chen, Yujun Shen, and Chun- hua Shen. Framer: Interactive video interpolation. In In- ternational Conference on Learning Representations (ICLR),

  10. [18]

    Anime2sketch: A sketch extractor for anime arts with deep networks

    Xiao Yang Yiheng Zhu Xiaohui Shen Xiaoyu Xiang, Ding Liu. Anime2sketch: A sketch extractor for anime arts with deep networks. https : / / github . com / Mukosame/Anime2Sketch, 2021. 4

  11. [19]

    Tooncrafter: Generative cartoon interpolation

    Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xin- tao Wang, Ying Shan, and Tien-Tsin Wong. Tooncrafter: Generative cartoon interpolation. ACM Transactions on Graphics (TOG), 43(6):1–11, 2024. 2, 3, 6

  12. [20]

    Dynamicrafter: Animating open-domain im- ages with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain im- ages with video diffusion priors. InComputer Vision – ECCV 2024, pages 399–417. Springer, 2024. 3

  13. [21]

    Colorizediffusion: Improv- ing reference-based sketch colorization with latent diffusion model

    Dingkun Yan, Liang Yuan, Erwin Wu, Yuma Nishioka, Is- sei Fujishiro, and Suguru Saito. Colorizediffusion: Improv- ing reference-based sketch colorization with latent diffusion model. In Proceedings of the Winter Conference on Applica- tions of Computer Vision (WACV), pages 509...

  14. [22]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an ...

  15. [23]

    Adding conditional control to text-to-image diffusion models, 2023

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2, 3

  16. [24]

    Magiccolor: Multi-instance sketch colorization

    Yinhan Zhang, Yue Ma, Bingyuan Wang, Qifeng Chen, and Zeyu Wang. Magiccolor: Multi-instance sketch colorization. CoRR, abs/2503.16948, 2025. 3

  17. [25]

    Sakuga-42m dataset: Scaling up cartoon research

    Yuxuan Mu Zhenglin Pan, Yu Zhu. Sakuga-42m dataset: Scaling up cartoon research. arXiv preprint arXiv:2405.07425, 2024. 4

  18. [26]

    Cobra: Efficient line art colorization with broader references

    Junhao Zhuang, Lingen Li, Xuan Ju, Zhaoyang Zhang, Chun Yuan, and Ying Shan. Cobra: Efficient line art colorization with broader references. CoRR, abs/2504.12240, 2025. 3 8

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.