REVIEW 4 major objections 5 minor 1 cited by
This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that a semantic segmentation network's mistakes can be detected by regenerating the image from its predicted labels and learning a dissimilarity metric between that reconstruction and the original input.
desk verdict An honest write-up of a plausible idea whose own experiments show it does not beat a softmax baseline; useful as a negative result, not as a positive claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-stage reconstruction-and-compare loop. A conditional GAN (pix2pix, in the paper's implementation) maps a semantic label map to a synthetic RGB image; the hope is that within the training domain, a correct label map gives a plausible reconstruction while an erroneous label map does not. The second stage is a patch-based dissimilarity detector whose feature extractor is the first seven layers of VGG16, chosen to keep the receptive field local, followed by one of three decision heads: 1x1 convolution with bilinear upsampling, deconvolutions, or fully connected layers. The detector is trained on triplets consisting of a real/synthetic patch pair and a hard negative sampled from a different synthetic image with sufficiently different class affiliation. The objective is $L(D) = \lambda_D \mathbb{E}_{t_i}[\log D(p_i^+)] + \mathbb{E}_{t_i}[\log(1 - D(p_i^-))]$, which the paper minimizes to make the detector assign high scores to inconsistent patch pairs.
What would settle it
Use a validation set with ground-truth labels, generate a synthetic image from those labels, and then corrupt a known region of the label map (for example, change road pixels to grass). If the learned dissimilarity detector does not score the corrupted region substantially higher than the uncorrupted region across many images, the central mechanism is not carrying the detection. The same test applied to the paper's own pipeline would isolate whether the low AUC stems from the generator, the detector, or the premise itself.
Extended reading notes
Core claim
The central claim is that a wrong semantic segmentation is visible as a visual inconsistency after the scene is regenerated from the labels. More precisely, the paper proposes passing the predicted segmentation map through a conditional GAN to synthesize an RGB image, then training a dissimilarity detector that compares local patches of the real image and the synthetic reconstruction. The detector is trained with triplets so it learns to recognize pairs that look alike versus pairs that differ, and it outputs a spatial map of suspicion rather than a single image-level score. The paper's own results show that the method can flag some out-of-distribution objects and misclassifications, but all three detector variants score lower area under the ROC curve (around 0.51–0.55) than the softmax-entropy baseline (0.73) on the out-of-distribution test, and the authors attribute the shortfall to the generator's failure to converge, its low output resolution, and the detector's sensitivity to brightness differences. The claim being argued is therefore conditional: the mechanism is sound in principle, but the generator and detector need to be substantially more reliable before the dissimilarity signal can serve as a trustworthy error detector.
Load-bearing premise
The entire pipeline rests on the assumption that a correct segmentation lets the generator redraw the image faithfully, so that a wrong segmentation stands out as a visual mismatch; in the paper's experiments this assumption did not hold because the conditional GAN failed to converge and produced low-resolution, brightness-sensitive reconstructions.
Editorial extensions
If this is right
- Any semantic segmentation network could be wrapped with this method to obtain per-pixel error and out-of-distribution detection without modifying or retraining the segmenter itself.
- Because the dissimilarity output is local and spatial, it can feed a downstream decision layer (for example, in autonomous driving) that needs to know where, not just whether, the segmentation is unreliable.
- The success of the method depends mostly on the generative model: a higher-resolution, stable conditional generator would directly improve detection, since the main failure mode in the experiments is the low quality and non-convergence of the generated images.
- The fully-connected detector head performs better on misclassification, while the deconvolution head performs better on out-of-distribution instances, suggesting that detector architecture should be chosen per task.
- In its current form the method is not competitive with the softmax-entropy baseline on global out-of-distribution detection; the paper positions it as a framework whose components require further work rather than as a finished detector.
Reading between the lines
- Because the method only compares an input with its reconstruction from a structured output, the same wrapper could be applied to other dense prediction tasks—depth estimation, instance segmentation, or panoptic maps—wherever a generator can redraw the scene from the prediction; the paper does not explore this.
- The systematic brightness sensitivity reported in Section 5 suggests an inexpensive remedy the authors do not test: aligning the global illumination of the generated image to the input before computing dissimilarity, which could remove the majority of false positives without retraining.
- A decisive comparison for the framework would use a modern, high-resolution conditional generator in place of pix2pix; if the gap to the entropy baseline persists under a near-perfect generator, the dissimilarity premise itself, rather than the implementation, is the limiting factor.
- The recurrence of low true-positive rates suggests that combining dissimilarity scores with the segmenter's own softmax entropy, rather than replacing it, might give the best of both: entropy for global confidence and dissimilarity for localizing the failures.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a wrapper for semantic segmentation error detection: given an input image, a segmentation network S predicts a semantic map, a conditional GAN (pix2pix) generates a synthetic RGB image from that map, and a learned dissimilarity detector D compares the synthetic image with the original input to localize out-of-distribution (OoD) or misclassified regions. Three detector decision architectures (resize, deconvolution, fully connected) are compared with a transfer-learning variant, the pretrained GAN discriminator, and a softmax-entropy baseline. Experiments on a 100-image subset of Mapillary Vistas (OoD plus misclassifications) and on Wilddash (misclassifications only) are reported, with the Wilddash evaluation being qualitative. The paper concludes that the method shows promise but has a high false-positive rate.
Significance. If the proposed pipeline worked as intended, it would provide a segmentation-network-agnostic way to localize semantic segmentation errors, which would be valuable for safety-critical applications such as autonomous driving. The paper also connects conditional image generation with local novelty detection, an interesting combination, and it honestly reports negative results. However, the evidence presented does not support the central claim: the best proposed method reaches an AUC of only 0.5469 on the main quantitative experiment, while a simple softmax-entropy baseline reaches 0.7256, and the authors themselves document that the cGAN did not converge and that the dissimilarity signal is dominated by brightness artifacts. These results, together with the absence of error bars and the purely qualitative misclassification experiment, mean the manuscript does not currently establish that visual dissimilarity is a useful signal for segmentation error detection.
major comments (4)
- [§4.1, Table 1] The quantitative results contradict the central claim. The three proposed dissimilarity detectors achieve AUC scores of 0.5469, 0.5466, and 0.5051 on the Mapillary Vistas subset, while the softmax entropy baseline achieves 0.7256. The proposed methods are therefore not only close to chance but are substantially worse than a standard baseline. Since the paper's thesis is that visual dissimilarity can detect wrongly segmented or OoD instances, this result is load-bearing and the manuscript does not provide any evidence that the proposed signal adds information over prediction entropy.
- [§4.2] The misclassification detection experiment on Wilddash is reported only qualitatively through example outputs. There is no AUC, F1 score, or comparison with the softmax entropy baseline for this experiment, so the claim that one architecture 'performs better' or 'shows promising results' is not supported by any quantitative evidence. A quantitative evaluation with a baseline is necessary to substantiate the method's utility for the misclassification-detection use case.
- [§5] The limitations stated in the Discussion invalidate the premise of the method as implemented. The paper states that the cGAN 'did not reach convergence', produces low-resolution images, and that the dissimilarity detector is so sensitive to brightness that slight illumination differences trigger detections even when semantics match, causing a very high false positive rate. This means the measured dissimilarity signal may reflect low-level reconstruction artifacts rather than semantic disagreement between the input and the predicted segmentation. The core assumption from §3.1, that a correct segmentation leads to a faithful generated image, is therefore not demonstrated, and the reported near-chance AUCs are consistent with the signal being dominated by artifacts rather than semantics.
- [§4.1] The evaluation uses only 100 images and reports no confidence intervals or statistical significance tests. Given that the differences among the proposed methods (e.g., 0.5469 vs. 0.5466) are far smaller than plausible sampling noise, the paper's comparative claims about detector architectures are not supported. Error bars or confidence intervals over multiple evaluation subsets are needed.
minor comments (5)
- [§3.2, Eq. (1)] The notation in Eq. (1) is unclear: D(p_i^+) is used both as a probability-like score and as the input to a logarithmic loss; the text should define whether D outputs a probability of dissimilarity or similarity, and how the positive/negative labels are assigned.
- [§3.4] The negative sampling procedure relies on semantic segmentations to determine whether two patches are 'semantically different'. This is a sensible design choice, but the threshold and the source of the semantic labels (ground truth versus predicted) should be specified precisely, since they directly affect the training signal.
- [§4.1] The paper states that 100 images containing 'boat' or 'snow' were extracted from Mapillary Vistas, but it does not report how many images contain each class, how many are from different cities, or how the subset was balanced. This information is needed to assess the representativeness of the evaluation.
- [§4.3, Figure 6] The loss-weighting experiment reports only a plot of F1 scores without numeric values or axis labels that are legible in the manuscript; exact F1 values and the corresponding thresholds should be stated in the text or table.
- [References] Reference [11] (Isola et al.) is missing the publication venue and year; other references are formatted inconsistently (e.g., some arXiv identifiers include years and others do not). The bibliography should be brought into a consistent style.
Circularity Check
No significant circularity: the dissimilarity detector is trained and evaluated on held-out data, and no target performance is fitted.
full rationale
The paper's derivation chain is: input image -> semantic segmentation S -> cGAN G conditioned on the segmentation -> synthetic RGB image -> learned dissimilarity detector D comparing original and synthetic images to localize OoD or misclassified regions. The detector is trained on triplets whose positive pairs are real/synthetic patches from the same location and whose negative pairs are synthetic patches from another image with sufficiently different class affiliation (Section 3.4). This training signal uses real/synthetic correspondence and semantic class differences; it does not use the held-out error masks or any target quantity that the paper later 'predicts'. The quantitative evaluation on Mapillary Vistas and Wilddash is performed on held-out data, and the reported AUC values (Table 1) come from that held-out comparison rather than from fitting. The paper's central premise — that a correct segmentation leads to a faithful reconstruction and an incorrect one to a detectable inconsistency — is indeed undermined by the paper's own Section 5 limitation statement: the cGAN 'did not reach convergence', produced low-resolution images, and brightness sensitivity caused high false positives. However, that is an empirical validity failure, not circularity: the premise is an assumption from which the method proceeds, not a quantity defined in terms of the method's output. There are no load-bearing self-citations: the cited prior works (pix2pix, AdapNet, DeepLabv3+, softmax entropy baseline) are standard external components and none is used to import the target result. No 'uniqueness theorem' from the authors' prior work is invoked, and no fitted parameter is renamed as a prediction. Therefore the derivation is self-contained and not circular.
Assumptions & free parameters
free parameters (2)
- lambda_D loss weighting =
1 (balanced)
- Negative sampling pixel-difference threshold =
not reported
assumptions (3)
- ad hoc to paper Reconstruction consistency is a valid signal for segmentation correctness.
- domain assumption A Cityscapes-trained cGAN produces faithful reconstructions of in-distribution images.
- domain assumption The dissimilarity detector transfers across datasets.
Cite this review
Pith. "Pith review of This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity." pith.science (2026). https://pith.science/paper/BDQSUTZ5
@misc{pith2026190900676,
author = {Pith},
title = {Pith review of: This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity},
year = {2026},
howpublished = {\url{https://pith.science/paper/BDQSUTZ5}},
note = {Machine review of arXiv:1909.00676}
}
read the original abstract
There has been a remarkable progress in the accuracy of semantic segmentation due to the capabilities of deep learning. Unfortunately, these methods are not able to generalize much further than the distribution of their training data and fail to handle out-of-distribution classes appropriately. This limits the applicability to autonomous or safety critical systems. We propose a novel method leveraging generative models to detect wrongly segmented or out-of-distribution instances. Conditioned on the predicted semantic segmentation, an RGB image is generated. We then learn a dissimilarity metric that compares the generated image with the original input and detects inconsistencies introduced by the semantic segmentation. We present test cases for outlier and misclassification detection and evaluate our method qualitatively and quantitatively on multiple datasets.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
From Pixel to Mask: A Survey of Out-of-Distribution Segmentation
A survey categorizing out-of-distribution segmentation methods for autonomous driving into test-time, outlier-exposure, reconstruction, and powerful-model families.
Reference graph
Works this paper leans on
-
[1]
A. P. Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recogni- tion, 30(7):1145–1159, 1997
work page 1997
-
[2]
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully con- nected crfs. IEEE transactions on pattern analysis and ma- chine intelligence, 40(4):834–848, 2018
work page 2018
-
[3]
L.-C. Chen, Y . Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for se- mantic image segmentation. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) , pages 801– 818, 2018
work page 2018
-
[4]
Q. Chen and V . Koltun. Photographic image synthesis with cascaded refinement networks. In IEEE International Con- ference on Computer Vision (ICCV), volume 1, page 3, 2017
work page 2017
-
[5]
Cordts, M
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016
2016
-
[6]
Y . Gal. Uncertainty in deep learning. University of Cam- bridge, 2016
work page 2016
-
[7]
Goodfellow, J
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014
2014
-
[8]
X. Han, T. Leung, Y . Jia, R. Sukthankar, and A. C. Berg. Matchnet: Unifying feature and metric learning for patch- based matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3279– 3286, 2015
2015
Show all 30 references
-
[9]
Harris and M
C. Harris and M. Stephens. A combined corner and edge detector. In Alvey vision conference, volume 15, pages 10–
-
[10]
Hendrycks and K
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural net- works. arXiv preprint arXiv:1610.02136, 2016
2016 arXiv
-
[11]
Isola, J.-Y
P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016
2016
-
[12]
Johnson, A
J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision , pages 694–711. Springer, 2016
2016
-
[13]
Kendall, V
A. Kendall, V . Badrinarayanan, and R. Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder- decoder architectures for scene understanding. arXiv preprint arXiv:1511.02680, 2015
2015 arXiv
-
[14]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[15]
K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence- calibrated classifiers for detecting out-of-distribution sam- ples. arXiv preprint arXiv:1711.09325, 2017
2017 arXiv
-
[16]
J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 3431–3440, 2015
2015
-
[17]
D. G. Lowe. Distinctive image features from scale- invariant keypoints. International journal of computer vi- sion, 60(2):91–110, 2004
2004
-
[18]
Neuhold, T
G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 4990–4999, 2017
2017
-
[19]
T. Pham, V . B. Kumar, T.-T. Do, G. Carneiro, and I. Reid. Bayesian semantic instance segmentation in open set world. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 3–18, 2018
2018
-
[20]
Pidhorskyi, R
S. Pidhorskyi, R. Almohsen, and G. Doretto. Genera- tive probabilistic novelty detection with adversarial autoen- coders. In Advances in Neural Information Processing Sys- tems, pages 6823–6834, 2018
2018
-
[21]
M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko. A review of novelty detection. Signal Processing, 99:215– 249, 2014
2014
-
[22]
Richter and N
C. Richter and N. Roy. Safe visual navigation via deep learn- ing and novelty detection. 2017
2017
-
[23]
Rippel, M
O. Rippel, M. Paluri, P. Dollar, and L. Bourdev. Metric learning with adaptive density discrimination.arXiv preprint arXiv:1511.05939, 2015
2015 arXiv
-
[24]
Schlegl, P
T. Schlegl, P. Seeb¨ock, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with gen- erative adversarial networks to guide marker discovery. In International Conference on Information Processing in Med- ical Imaging, pages 146–157. Springer, 2017
2017
-
[25]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
2014 arXiv
-
[26]
Valada, J
A. Valada, J. Vertens, A. Dhall, and W. Burgard. Adap- net: Adaptive semantic segmentation in adverse environmen- tal conditions. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 4644–4651. IEEE, 2017
2017
-
[27]
H.-g. Wang, X. Li, and T. Zhang. Generative adversarial net- work based novelty detection usingminimized reconstruction error. Frontiers of Information Technology & Electronic En- gineering, 19(1):116–125, 2018
2018
-
[28]
J. Wang, Y . Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y . Wu. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1386–1393, 2014
2014
-
[29]
Zagoruyko and N
S. Zagoruyko and N. Komodakis. Learning to compare im- age patches via convolutional neural networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4353–4361, 2015
2015
-
[30]
Zendel, K
O. Zendel, K. Honauer, M. Murschitz, D. Steininger, and G. Fernandez Dominguez. Wilddash-creating hazard-aware benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402–416, 2018. Figure 7. Qualitative Examples from the Error Detection experime...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.