Image Colorization By Capsule Networks

G\"okhan \"Ozbulak

REVIEW 3 major objections 6 minor 17 references

Image Colorization By Capsule Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A shallow capsule network, ColorCapsNet, adapted from the original CapsNet with VGG-19 features and MSE loss, produces automatic colorization results comparable to NTIRE 2019 challenge participants on DIV2K.

desk verdict A modest CapsNet colorization adaptation that honestly reports routing has no effect, but never shows the capsule machinery matters. read the letter →

arxiv 1908.08307 v1 pith:BL37RLUJ submitted 2019-08-22 eess.IV cs.CV

Gökhan Özbulak This is my paper · ORCID

classification eess.IVcs.CV

keywords imagecolorizationcapsulenetworkrouting-by-agreementCIELabcolorspaceVGG-19transferlearningDIV2KPSNR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to show that capsule networks, originally designed for classification, can be turned into automatic image colorizers. The proposed ColorCapsNet maps grayscale image patches to CIE Lab color patches using a shallow encoder-decoder capsule topology, VGG-19 pretrained features, batch normalization, and mean squared error loss. Evaluated on DIV2K, it reports validation PSNR of 22.20 and SSIM of 0.88 at 50 epochs, and test PSNR of 21.08 and SSIM of 0.85 at 10 epochs, placing it among the comparable entries in the NTIRE 2019 colorization challenge. The significance would be that capsule routing, not just convolutions, can support image-to-image generation tasks.

What carries the argument

The central object is ColorCapsNet, an autoencoder-like capsule network in which a capsule is a group of neurons whose output vector encodes an entity's instantiation parameters. The encoder starts with the first two convolutional layers of VGG-19 (with transferred weights) followed by batch normalization, a primary capsule layer, and a capsule layer with six capsules. Dynamic routing-by-agreement transfers information between capsule layers and is intended to segment objects internally; the decoder then reconstructs the $a$ and $b$ chrominance channels from the capsule vectors, with the $L$ channel taken from the grayscale input. Patch size and routing iterations are treated as critical hyperparameters, with 9x9 patches and one routing iteration chosen.

What would settle it

Train ColorCapsNet on DIV2K with routing iterations set to zero or with the capsule layer replaced by a plain convolutional layer of similar size, keeping VGG-19 features and MSE loss identical; if validation PSNR and SSIM remain at roughly 22.20 and 0.88, the central role claimed for routing is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the generative and segmentation capabilities of the original CapsNet carry over to colorization when the feature detector is replaced with the first two convolutional layers of VGG-19 (initialized with pretrained weights), batch normalization is added, the number of class capsules is reduced to six, and margin loss is exchanged for MSE in CIE Lab space. The resulting ColorCapsNet is patch-based and fully automatic, and it achieves the reported PSNR/SSIM values on DIV2K validation and test sets. The authors interpret this as evidence that capsule networks deserve further investigation for colorization.

Load-bearing premise

The load-bearing premise is that capsule routing-by-agreement, rather than the transferred VGG-19 features and the decoder, is what gives ColorCapsNet its colorization ability; the paper calls routing one of the most critical hyperparameters but offers no ablation that removes or replaces routing to prove that.

Editorial extensions

If this is right

If ColorCapsNet's results hold, capsule networks are viable for image generation tasks beyond classification, not just for object recognition.
The reported scores imply that a shallow capsule architecture, combined with transferred VGG-19 features, can compete with more complex automatic colorization models on a standard benchmark.
Because the network is patch-based and trained in CIE Lab space, the method suggests that local color distribution learning with MSE loss is sufficient for plausible colorization in this setting.
The authors' observation that loss curves had not plateaued at 50 epochs implies that longer training on DIV2K may push the scores higher, and the paper explicitly leaves this as an open direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate routing's contribution: since changing routing iterations from 1 to 3 barely changed train loss, a plausible inference is that the VGG-19 features and the decoder, not the routing-by-agreement mechanism, drive most of the colorization quality.
A direct testable extension would replace the capsule decoder with a plain convolutional decoder of similar capacity; if PSNR/SSIM stay near 22.20/0.88, capsule routing is not load-bearing for this task.
The same architecture could be extended to multimodal colorization by predicting a per-pixel color distribution instead of a single Lab value, which would address the ill-posed ambiguity the paper acknowledges.
The observed invariance to routing iterations suggests that dynamic routing may matter more for classification-type tasks than for dense prediction; this is an inference beyond the paper's claims.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a simple application paper: it takes the original CapsNet, swaps the front-end for VGG-19's first two conv layers, changes margin loss to MSE, and trains it patch-wise for colorization on DIV2K. That's the whole novelty, and it's fine as a modest proof-of-concept. The paper does a couple of honest things: it shows in Figure 4b that the number of routing iterations doesn't affect the loss, then picks one iteration for speed; and it openly says the test model was fine-tuned on the DIV2K validation set. Both admissions are to its credit.

The soft spots are the ones the stress-test note names. There is no comparison to a non-capsule control. If you take the same VGG-19 features, a latent vector of the same size, and MSE loss, a plain convolutional decoder would likely do as well or better. Without that ablation, the title's implication—that capsule mechanisms drive the colorization—is unsubstantiated. The routing claim is especially shaky: Section 3.4 calls routing 'one of the most critical hyperparameters,' but Figure 4b says changing it does nothing. The paper doesn't resolve that tension. 'Comparable' also isn't established: Table 1 only gives the challenge's aggregate worst and best, not named baselines, and the validation-set fine-tuning confounds the test numbers. Code is not provided, so the numbers can't be checked.

None of this is fatal in the sense of the paper being nonsense. It's a preliminary empirical report. But the central claim is overreach: the evidence doesn't show ColorCapsNet's capsule-specific machinery is responsible for the observed PSNR/SSIM. A serious referee should ask for ablations against a convolutional decoder, named baselines or at least a proper held-out protocol, and a discussion of why routing appears to have no effect.

I'd send this to peer review rather than desk-reject, because the question is legitimate and the paper is readable and honest in its reporting. The likely outcome is major revision or a workshop-level paper. My own verdict is skeptical: I wouldn't cite it for the capsule claim, but I'd maybe bring it up in a reading group as an example of an under-ablated application paper.

Referee Report

3 major / 6 minor

Summary. The paper proposes ColorCapsNet, an adaptation of the original CapsNet to the task of automatic image colorization. The architecture uses a VGG-19-based feature extractor, a primary capsule layer, and a capsule classification layer, with MSE loss in CIE Lab space. The method is trained on ILSVRC 2012 and DIV2K and evaluated on NTIRE 2019 colorization challenge data. The authors report a validation PSNR of 22.20 and SSIM of 0.88 after 50 epochs, and a test PSNR of 21.08 and SSIM of 0.85 after 10 epochs, and claim these results are promising and comparable.

Significance. The application of capsule networks to colorization is a novel and potentially interesting direction. The paper provides a clear description of the architecture and a comparative analysis of patch size, routing iterations, and topology choices. However, the central claim that capsule-specific mechanisms (routing-by-agreement and capsule representations) contribute to the colorization quality is not supported by the evidence: Figure 4b shows no effect of routing iterations, and no non-capsule control is reported. The evaluation protocol also confounds the test numbers by fine-tuning on the validation set. If the authors add a proper ablation and a standardized comparison, the work could be a useful contribution to the community. The manuscript is honest about the observed lack of routing sensitivity.

major comments (3)

[Section 4.1; Table 1] The test evaluation is confounded by fine-tuning on the validation set. Section 4.1 states that after training on the DIV2K train split, the network is "finally trained on this validation set to colorize the test images," so the test numbers in Table 1 do not follow the standardized NTIRE evaluation protocol and are not directly comparable to the listed worst/best challenge values. Additionally, the validation and test rows come from different training durations (50 vs 10 epochs), so the two numbers should not be presented as a single model's performance.
[Section 3.4; Figure 4b] The paper's central claim that routing-by-agreement contributes to colorization is contradicted by its own Figure 4b, which shows no effect of varying the number of routing iterations. No ablation is provided that replaces the capsule decoder with a plain convolutional decoder using the same VGG-19 front-end, latent dimensionality, patch size, and MSE loss. Without such a control, the reported PSNR/SSIM values cannot be attributed to capsule mechanisms; they may arise entirely from the pretrained VGG-19 features and the MSE objective. This is a load-bearing gap for the title and Section 3.4 claims.
[Table 1] The comparison to "Worst" and "Best PSNR/SSIM" from the NTIRE 2019 challenge is not a comparison to named baselines. Since the worst and best values may come from different participants for each metric, the claim that ColorCapsNet has "comparable colorization performance" is not quantitatively supported. The authors should cite specific competing methods from the NTIRE report and evaluate under the identical protocol, including the same training epochs and no validation-set fine-tuning.

minor comments (6)

[Section 1] There is a typo in Section 1: "These studies may be classiﬁed as in two categories:" should read "These studies may be classified into two categories:".
[Equation (6)] Equation (6) is ambiguous: it writes "1/YX" while the summation bounds are over y=1..Y and x=1..X; this should be written as 1/(XY) or with an explicit double sum.
[Figure 4(c)] The caption "the train loss decreases with the patch size goes down exponentially" is grammatically incomplete, and the figure does not use a logarithmic scale, so the word "exponentially" is not supported by the displayed data.
[Section 4.1] The description of DIV2K says the train set has 794 grayscale and 800 RGB images, and then "794 out of 800 images in the train set are used"; the reason for excluding six images is not explained.
[Section 3.1/3.2] The paper does not specify whether patches are sampled densely or randomly, nor the batch size used for training; these details are needed for reproduction.
[Table 1] The footnote markers "1" and "2" after the ColorCapsNet numbers are not formatted as superscripts, which makes the table harder to read.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all reported results are empirical evaluations against external benchmarks (DIV2K, NTIRE Challenge), with no fitted parameter renamed as a prediction and no load-bearing self-citations.

full rationale

The paper proposes ColorCapsNet, an adaptation of CapsNet for image colorization, and evaluates it on DIV2K with PSNR/SSIM metrics. The central claims are empirical: the network is trained on ILSVRC 2012 and DIV2K, and results are compared to NTIRE 2019 challenge best/worst values. No equation in the paper defines a predicted quantity in terms of the fitted parameters that allegedly predict it. The loss function (MSE), the metrics (PSNR, SSIM), and the architecture components (CapsNet, VGG-19 features, batch normalization) are all standard and are not derived from the reported outcome. The paper includes no self-citations, no uniqueness theorems from the authors' prior work, and no ansatz smuggled in via citation. The unsupported claim that routing-by-agreement contributes to segmentation is a weakness in experimental validation (no ablation with a non-capsule decoder; Figure 4b even shows routing has no effect), but this is a correctness or evidence concern, not circularity: the reported PSNR/SSIM numbers are not forced by the choice of routing iterations or by any self-referential definition. The fine-tuning on the DIV2K validation set before testing is an unusual protocol but does not make the test numbers a fitted rename of training data. Therefore, no specific circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its central claim rests on a few domain assumptions about transfer learning and capsule routing; notably the routing benefit is not ablated and is even contradicted by the observation that routing count has no effect.

free parameters (3)

number of class capsules = 6
Reduces trainable parameters by ~2.39M; loss roughly unchanged (Figure 4a). Chosen by hand, not by validation performance.
number of routing iterations = 1
Claimed no effect on performance (Figure 4b); selected to speed up training.
patch size = 9x9
Chosen because training loss decreases with smaller patch size (Figure 4c); this is fitting to training loss, and smaller patches limit context for colorization.

assumptions (5)

domain assumption The original CapsNet's generative and segmentation capabilities can be transferred to image colorization by changing the loss and output space.
Central to the paper's motivation; not proven by the experiments, which do not isolate this effect.
domain assumption CIE Lab colorspace with L2/MSE loss is appropriate for colorization.
Taken from [17]; accepted as standard.
standard math Batch normalization reduces internal covariate shift and improves convergence.
Standard result from [9], not re-derived.
domain assumption Pretrained VGG-19 weights provide useful low-level features for colorization.
Asserted in Section 3.2; only train loss evidence given.
standard math PSNR and SSIM are valid metrics for colorization quality.
Standard image quality metrics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Image Colorization By Capsule Networks." pith.science (2026). https://pith.science/paper/BL37RLUJ

@misc{pith2026190808307,
  author       = {Pith},
  title        = {Pith review of: Image Colorization By Capsule Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BL37RLUJ}},
  note         = {Machine review of arXiv:1908.08307}
}

read the original abstract

In this paper, a simple topology of Capsule Network (CapsNet) is investigated for the problem of image colorization. The generative and segmentation capabilities of the original CapsNet topology, which is proposed for image classification problem, is leveraged for the colorization of the images by modifying the network as follows:1) The original CapsNet model is adapted to map the grayscale input to the output in the CIE Lab colorspace, 2) The feature detector part of the model is updated by using deeper feature layers inherited from VGG-19 pre-trained model with weights in order to transfer low-level image representation capability to this model, 3) The margin loss function is modified as Mean Squared Error (MSE) loss to minimize the image-to-imagemapping. The resulting CapsNet model is named as Colorizer Capsule Network (ColorCapsNet).The performance of the ColorCapsNet is evaluated on the DIV2K dataset and promising results are obtained to investigate Capsule Networks further for image colorization problem.

Figures

Figures reproduced from arXiv: 1908.08307 by the authors.

**Figure 2.** Data pre-processing flow. 3.2. Network Design The first modification is on the feature detection part of the network. In the original CapsNet [14], this part consists of one convolutional layer with 256 filters. These filters have size of 9x9 with stride of 1 and are activated with Rectified Linear Unit (ReLU) to feed the feature representation layer named as primary capsule. In proposed method, the feature detec… view at source ↗

**Figure 3.** The ColorCapsNet topology. encoder and decoder networks and it represents the hidden variables related with colorization task. 3.3. Optimization Procedure In the original CapsNet model, the margin loss is used to minimize the loss during training. The margin loss is defined for each capsule in the classification layer (capsule layer) as below: Lc = Tcmax(0, 0.9−kvck) 2+λ(1−Tc)max(0, kvck−0.1)2 (4) where λ = 0.5, vc … view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Comparative analysis for network design of the ColorCapsNet. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: The train loss performance of the ColorCapsNet. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Some colorization results from DIV2K dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Colorization progress across training. References [1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–13…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

[1]

Ntire 2017 challenge on single image super-resolution: Dataset and study

Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–135, 2017

work page 2017
[2]

Automatic image colorization via multimodal predictions

Guillaume Charpiat, Matthias Hofmann, and Bernhard Sch¨olkopf. Automatic image colorization via multimodal predictions. In European conference on computer vision , pages 126–139. Springer, 2008

work page 2008
[3]

Deep col- orization

Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep col- orization. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 415–423, 2015

work page 2015
[4]

Se- mantic colorization with internet images

Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu- Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. Se- mantic colorization with internet images. In ACM Transac- tions on Graphics (TOG), volume 30, page 156. ACM, 2011

work page 2011
[5]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

2009
[6]

Ntire 2019 challenge on image colorization: Report

Shuhang Gu, Radu Timofte, Richard Zhang, et al. Ntire 2019 challenge on image colorization: Report. In The IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019

work page 2019
[7]

Image colorization us- ing similar images

Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. Image colorization us- ing similar images. In Proceedings of the 20th ACM inter- national conference on Multimedia , pages 369–378. ACM, 2012

work page 2012
[8]

Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simulta- neous classiﬁcation

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simulta- neous classiﬁcation. ACM Transactions on Graphics (TOG), 35(4):110, 2016

work page 2016

Show all 17 references

[9]

Batch normalization: Accelerating deep network training by reducing internal co- variate shift

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. arXiv preprint arXiv:1502.03167, 2015

2015 arXiv
[10]

Col- orization by example

Revital Ironi, Daniel Cohen-Or, and Dani Lischinski. Col- orization by example. In Rendering Techniques, pages 201–
[11]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014

2014 arXiv
[12]

Learning representations for automatic colorization

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European Conference on Computer Vision, pages 577–593. Springer, 2016

2016
[13]

Colorization using optimization

Anat Levin, Dani Lischinski, and Yair Weiss. Colorization using optimization. In ACM transactions on graphics (tog), volume 23, pages 689–694. ACM, 2004

2004
[14]

Dy- namic routing between capsules

Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dy- namic routing between capsules. In Advances in neural in- formation processing systems, pages 3856–3866, 2017

2017
[15]

Very deep convo- lutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

2014 arXiv
[16]

Fast image and video colorization using chrominance blending

Liron Yatziv and Guillermo Sapiro. Fast image and video colorization using chrominance blending. IEEE transactions on image processing, 15(5):1120–1129, 2006

2006
[17]

Colorful image colorization

Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649–666. Springer, 2016

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Ntire 2017 challenge on single image super-resolution: Dataset and study

Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–135, 2017

work page 2017

[2] [2]

Automatic image colorization via multimodal predictions

Guillaume Charpiat, Matthias Hofmann, and Bernhard Sch¨olkopf. Automatic image colorization via multimodal predictions. In European conference on computer vision , pages 126–139. Springer, 2008

work page 2008

[3] [3]

Deep col- orization

Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep col- orization. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 415–423, 2015

work page 2015

[4] [4]

Se- mantic colorization with internet images

Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu- Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. Se- mantic colorization with internet images. In ACM Transac- tions on Graphics (TOG), volume 30, page 156. ACM, 2011

work page 2011

[5] [5]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

2009

[6] [6]

Ntire 2019 challenge on image colorization: Report

Shuhang Gu, Radu Timofte, Richard Zhang, et al. Ntire 2019 challenge on image colorization: Report. In The IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019

work page 2019

[7] [7]

Image colorization us- ing similar images

Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. Image colorization us- ing similar images. In Proceedings of the 20th ACM inter- national conference on Multimedia , pages 369–378. ACM, 2012

work page 2012

[8] [8]

Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simulta- neous classiﬁcation

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simulta- neous classiﬁcation. ACM Transactions on Graphics (TOG), 35(4):110, 2016

work page 2016