REVIEW 4 major objections 6 minor 30 references
Feedbackward Decoding for Semantic Segmentation
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper proposes feedbackward decoding: after a convolutional encoder processes an image forward, the same network runs backward to recover spatial resolution, with the encoder's weights reused as the decoder's weights.
desk verdict Neat parameter-reuse idea, but the experimental support is too uncontrolled to back the performance claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the transposed weight tensor of a convolutional layer, formed by swapping the input-channel and output-channel dimensions of $W$ rather than by transposing spatial dimensions as in transposed convolution. This gives $W'$ with the same size and the same filters, preserving the inner structure of the original filters while changing which channel grouping they feed. In the forward pass each column of the filter matrix produces one output channel; in the backward pass each former row becomes a column, so the layer can expand or contract channel counts in the opposite direction. The network's effective depth is then counted by how many times information passes through a convolutional layer, which is why the paper describes its 14-layer model as having comparable capacity to the 26-convolutional-layer SegNet.
What would settle it
Hold the training pipeline fixed and replace the transposed encoder weights in the backward pass with randomly initialized weights of identical shape; if the random decoder matches or beats the transposed-weight decoder on Cityscapes IoU class, then the claimed benefit of reusing encoder weights is refuted. A more direct check is per-layer round-trip reconstruction: encoding a batch of images and decoding with the transposed weights should reconstruct layer inputs far better than decoding with random weights, and that margin should grow with the final segmentation score.
Extended reading notes
Core claim
The central claim is that a convolutional layer is inherently bidirectional. Because the weight tensor for mapping $c_1$ input channels to $c_2$ output channels has exactly the same size as the tensor for the reverse mapping, $W' \in \mathbb{R}^{m\times n\times c_2\times c_1}$ can be obtained from $W \in \mathbb{R}^{m\times n\times c_1\times c_2}$ by permuting the channel dimensions. Grouping filters into columns in the forward pass becomes grouping them into rows in the backward pass, so the original trained filters are kept intact while their role is reversed. The paper implements this on VGG-16, using nearest-neighbor interpolation where max pooling occurred, summing forward and backward feature maps at matching depths, and adding one classification layer. Its best variant, VGG-PWN, reaches 67.3 IoU class on Cityscapes, the highest among the compared VGG-16-based models, with only 14.7M parameters.
Load-bearing premise
The load-bearing premise is that a convolution's weight tensor, after swapping its input and output channels, provides a genuinely useful decoding projection: that the same filters which extract semantic features forward can meaningfully map those features back toward spatial detail.
Editorial extensions
If this is right
- Semantic segmentation can be built as an encoder plus one tiny classification layer, with no upsampling decoder that has its own parameters.
- Reusing encoder weights in the backward pass roughly halves the parameter count relative to a symmetric encoder-decoder such as SegNet, at comparable effective depth.
- Decoder weights no longer have to be trained from scratch, so training can work with many fewer labeled images; the paper's model used only the 2,975 fully labeled Cityscapes training images, while DeepLab LargeFOV used 19,998 extra weakly labeled samples.
- Permuting weights only when the channel dimension changes, as in the PWN variant, gives better results than permuting at every layer as in PWP, indicating that preserving existing filter groups matters for the backward projection.
- Because the encoder's structure is unchanged, the method can be applied to other base networks rather than only VGG-16.
Reading between the lines
- If transposed encoder weights genuinely act as an approximate inverse projection, then encouraging orthogonality or spectral balance in encoder filters should improve feedbackward decoding; this is testable by regularizing layer weights and measuring segmentation accuracy.
- The method predicts that the round-trip reconstruction error of a forward-backward pass should be much smaller for trained encoder weights than for random weights of the same shape, and that this gap should correlate with final segmentation accuracy.
- The same weight-reuse trick could be tried on other dense prediction tasks, such as depth estimation or edge detection, where an encoder must recover spatial detail from high-level features without a separately trained decoder.
- The PWN-versus-PWP comparison implies that arbitrary permutations can break the team structure of filters; a further extension would be to learn which channel permutations to apply in the backward pass rather than choosing them only when necessary.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes 'feedbackward decoding' for semantic segmentation: instead of attaching a separate decoder, the encoder's own convolutional layers are used in reverse, with weight tensors derived by permuting the input and output channel dimensions of the original convolution kernels. The authors adapt VGG-16 by keeping its 13 convolutional layers, adding one classification layer, and adding two batch-normalization operations per layer, one for each pass direction. On the Cityscapes benchmark they report that their best variant, VGG-PWN, reaches 67.3 IoU class while using 14.7M parameters, outperforming published scores for SegNet, DeepLab LargeFOV, and FCN-8s with fewer parameters. The paper argues that this demonstrates the encoder itself can serve as the decoder, and that the permutation operation preserves useful filter structure.
Significance. If the empirical claims hold, the idea is significant: it offers a way to construct a decoder without training additional decoding layers, reducing parameter count and potentially improving generalization by reusing pretrained features. The mathematical observation in Section 3, that a convolution weight tensor can be transposed by swapping channel dimensions to reverse the direction of a convolution, is correct and is a clean, useful conceptual contribution. The paper also makes a falsifiable quantitative claim about Cityscapes. However, the experimental support is currently not strong enough to substantiate the central claim: the comparisons are against published baseline numbers rather than re-run baselines, the contribution of the permutation mechanism is not isolated by ablations, and all results come from a single training run. The promise of public code is not fulfilled in the manuscript, further limiting reproducibility.
major comments (4)
- [§5, Table 1] The central experimental claim rests entirely on comparisons with scores taken from previously published papers for SegNet, DeepLab LargeFOV, and FCN-8s, rather than on re-running those models under the protocol described in Section 4. Section 6 concedes that DeepLab LargeFOV was trained with 19,998 additional weakly labeled samples, while VGG-PWN uses only the 2,975 fully labeled Cityscapes training images. Because training data, loss function, augmentation, input resolution, and evaluation protocol all differ, the reported margins of roughly 2-3 IoU class points in Table 1 cannot be attributed to feedbackward decoding. The authors should re-run the baseline models under identical conditions or substantially weaken the comparative claims.
- [§4, Figures 1-2 and §6] No ablation isolates the permuted-weight encoder-as-decoder mechanism. The backward pass includes skip additions, two additional batch-normalization operations per layer, and 27 effective convolutional passes, so a conventional decoder with the same parameter and computation budget could plausibly produce the same or better results without reusing the encoder weights through the Section 3 permutation. Please include controls such as an untied decoder trained from scratch, a decoder with the same topology but without weight reuse, and a tied decoder using nontransposed or randomly projected weights.
- [§4 and §5] All reported numbers come from a single training run with early stopping on the validation set, and no seeds or error bars are reported. Given that the advantage over DeepLab LargeFOV and FCN-8s is only 1.9-2.5 IoU class points, run-to-run variance could erase those margins. The authors should report mean and standard deviation over multiple seeds, or at least two complete runs, and state the precise early-stopping rule.
- [§3] The derivation in Section 3 establishes dimensional compatibility of the permuted weight tensor W', but it does not explain why pretrained filters should remain useful as decoding projections after permutation. This is the core assumption of the method, and it is currently supported only by the unablated experiments. A direct test, such as comparing against a randomly initialized or learned decoder of the same topology, or an analytical argument about filter structure preservation, would make the claim load-bearing rather than purely empirical.
minor comments (6)
- [§4] The statement 'No other layers are added' is misleading because each convolutional layer receives two additional batch-normalization operations, one for the forward pass and one for the backward pass; please clarify that no additional decoder layers are added, rather than no layers at all.
- [§3] In the paragraph defining W', the text writes W' ∈ R^{h×w×c2×c1}, but the original weight tensor was defined as W ∈ R^{m×n×c1×c2}; the spatial dimensions should be m and n, not h and w.
- [§4] Training details are incomplete: the number of epochs, the exact early-stopping criterion, batch-normalization momentum and epsilon, whether batch-normalization statistics are computed separately for the forward and backward passes, and the precise handling of pooling indices in the backward pass are not reported.
- [Figures 1 and 2] The difference between VGG-PWP and VGG-PWN is not visually evident from the figures, and the captions do not explain the permutation policy; please annotate the figures or add a table that clearly marks which layers permute their weights in the backward pass.
- [Abstract and §7] The abstract and conclusion both promise 'Our code will be publicly available later,' but no code or repository link is provided; please include an actual link or state the release plan in the final version, since the experiments are otherwise difficult to reproduce.
- [Table 1] The rows for SegNet, DeepLab LargeFOV, and FCN-8s should cite the specific source papers and, if available, the exact evaluation configuration (e.g., training set size and post-processing) so that readers can assess comparability.
Circularity Check
No circularity; the feedbackward decoding construction is an architectural proposal whose performance is evaluated on an external benchmark.
full rationale
The paper's central construction is a weight permutation: Section 3 defines W' by swapping the input and output channel dimensions of W, and Section 4 measures Cityscapes IoU after training. No fitted parameter is renamed as a prediction, and no claimed result is entailed by the definition of W'. The decoder uses W' derived from W, but the empirical claim is that this construction yields competitive segmentation accuracy, which is checked against an external benchmark in Table 1. The comparisons use published numbers for SegNet, DeepLab LargeFOV, and FCN-8s rather than re-run baselines; that is an experimental-control weakness and a correctness risk, not circularity, because those numbers are not inputs to the model and do not force the reported IoU values. The paper contains no self-citation chain, no imported uniqueness theorem, and no fitted-input-call-prediction pattern. Section 6's explanation that VGG-PWN outperforms VGG-PWP because permuting only when necessary preserves filter groupings is a post-hoc empirical observation, not a derivation from the permutation rule itself. Thus the derivation chain is self-contained: the construction is defined from the existing encoder weights, and the contribution is evaluated externally rather than being equivalent to its own inputs.
Assumptions & free parameters
assumptions (2)
- standard math A convolutional layer is a linear operation whose transpose (weight permutation) maps features back in the opposite direction.
- domain assumption Published Cityscapes scores for SegNet, DeepLab LargeFOV, and FCN-8s are directly comparable to the authors' own runs despite differences in training data and protocol.
Cite this review
Pith. "Pith review of Feedbackward Decoding for Semantic Segmentation." pith.science (2026). https://pith.science/paper/P4UVIEGM
@misc{pith2026190808584,
author = {Pith},
title = {Pith review of: Feedbackward Decoding for Semantic Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/P4UVIEGM}},
note = {Machine review of arXiv:1908.08584}
}
read the original abstract
We propose a novel approach for semantic segmentation that uses an encoder in the reverse direction to decode. Many semantic segmentation networks adopt a feedforward encoder-decoder architecture. Typically, an input is first downsampled by the encoder to extract high-level semantic features and continues to be fed forward through the decoder module to recover low-level spatial clues. Our method works in an alternative direction that lets information flow backward from the last layer of the encoder towards the first. The encoder performs encoding in the forward pass and the same network performs decoding in the backward pass. Therefore, the encoder itself is also the decoder. Compared to conventional encoder-decoder architectures, ours doesn't require additional layers for decoding and further reuses the encoder weights thereby reducing the total number of parameters required for processing. We show by using only the 13 convolutional layers from VGG-16 plus one tiny classification layer, our model significantly outperforms other frequently cited models that are also adapted from VGG-16. On the Cityscapes semantic segmentation benchmark, our model uses 50.0% less parameters than SegNet and achieves an 18.1% higher "IoU class" score; it uses 28.3% less parameters than DeepLab LargeFOV and the achieved "IoU class" score is 3.9% higher; it uses 89.1% fewer parameters than FCN-8s and the achieved "IoU class" score is 3.1% higher. Our code will be publicly available on Github later.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Segnet: A deep convolutional encoder-decoder architecture for image segmentation
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. CoRR, abs/1511.00561, 2015. URL http://arxiv.org/abs/1511.00561
arXiv 2015
-
[3]
Brostow, Julien Fauqueur, and Roberto Cipolla
Gabriel J. Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30 0 (2): 0 88 -- 97, 2009. ISSN 0167-8655. doi:https://doi.org/10.1016/j.patrec.2008.04.005. URL http://www.sciencedirect.com/science/article/pii/S0167865508001220. Video-based Object and Event Analysis
-
[4]
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR, abs/1412.7062, 2014
arXiv 2014
-
[5]
Liang - Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. CoRR, abs/1606.00915, 2016. URL http://arxiv.org/abs/1606.00915
arXiv 2016
-
[6]
Rethinking atrous convolution for semantic image segmentation
Liang - Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017. URL http://arxiv.org/abs/1706.05587
arXiv 2017
-
[7]
Encoder-decoder with atrous separable convolution for semantic image segmentation
Liang - Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. CoRR, abs/1802.02611, 2018. URL http://arxiv.org/abs/1802.02611
arXiv 2018
-
[8]
Xception: Deep learning with depthwise separable convolutions
Fran c ois Chollet. Xception: Deep learning with depthwise separable convolutions. CoRR, abs/1610.02357, 2016. URL http://arxiv.org/abs/1610.02357
arXiv 2016
Show all 30 references
-
[9]
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CoRR, abs/1604.01685, 2016. URL http://arxiv.org/abs/1604.01685
2016 arXiv
-
[10]
A guide to convolution arithmetic for deep learning
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning. ArXiv, abs/1603.07285, 2016
2016 arXiv
-
[11]
Deep Learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org
2016
-
[12]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385
2015 arXiv
-
[13]
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL http://arxiv.org/abs/1502.03167
2015 arXiv
-
[14]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014
2014 arXiv
-
[15]
Guosheng Lin, Anton Milan, Chunhua Shen, and Ian D. Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. CoRR, abs/1611.06612, 2016. URL http://arxiv.org/abs/1611.06612
2016 arXiv
-
[16]
Recent progress in semantic image segmentation
Xiaolong Liu, Zhidong Deng, and Yuhan Yang. Recent progress in semantic image segmentation. CoRR, abs/1809.10198, 2018. URL http://arxiv.org/abs/1809.10198
2018 arXiv
-
[17]
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. CoRR, abs/1411.4038, 2014. URL http://arxiv.org/abs/1411.4038
2014 arXiv
-
[18]
Balak, Pavel Karas, Tereza Bolcková, Markéta Štreitová, Craig Carthel, Stefano Coraluppi, Nathalie Harder, Karl Rohr, Klas E
Martin Maška, Vladimír Ulman, David Svoboda, Pavel Matula, Petr Matula, Cristina Ederra, Ainhoa Urbiola, Tomás España, Subramanian Venkatesan, Deepak M.W. Balak, Pavel Karas, Tereza Bolcková, Markéta Štreitová, Craig Carthel, Stefano Coraluppi, Nathalie Harder, Karl Rohr, Klas...
2014
-
[19]
The natural language decathlon: Multitask learning as question answering
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730, 2018. URL http://arxiv.org/abs/1806.08730
2018 arXiv
-
[20]
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp.\ 807--814, USA, 2010. Omnipress. ISBN 978-1-60558-907-7. URL http:...
2010
-
[21]
The mapillary vistas dataset for semantic understanding of street scenes
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bul\`o, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In International Conference on Computer Vision (ICCV), 2017. URL https://www.mapillary.com/dataset/vistas
2017
-
[22]
Deconvolution and checkerboard artifacts
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 2016. doi:10.23915/distill.00003. URL http://distill.pub/2016/deconv-checkerboard
2016 doi
-
[23]
Optimizing intersection-over-union in deep neural networks for image segmentation
Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-union in deep neural networks for image segmentation. In George Bebis, Richard Boyle, Bahram Parvin, Darko Koracin, Fatih Porikli, Sandra Skaff, Alireza Entezari, Jianyuan Min, Daisuke Iwai, Amela Sadagic, Carlos Sch...
2016
-
[24]
YOLO9000: better, faster, stronger
Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. CoRR, abs/1612.08242, 2016. URL http://arxiv.org/abs/1612.08242
2016 arXiv
-
[25]
Yolov3: An incremental improvement
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/1804.02767, 2018. URL http://arxiv.org/abs/1804.02767
2018 arXiv
-
[26]
Girshick, and Ali Farhadi
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. CoRR, abs/1506.02640, 2015. URL http://arxiv.org/abs/1506.02640
2015 arXiv
-
[27]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04597, 2015. URL http://arxiv.org/abs/1505.04597
2015 arXiv
-
[28]
Bernstein, Alexander C
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei - Fei Li. Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575, 2014. URL http://ar...
2014 arXiv
-
[29]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014
2014 arXiv
-
[30]
Multi-scale context aggregation by dilated convolutions
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. CoRR, abs/1511.07122, 2015
2015 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.