Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that a semantic segmentation network's mistakes can be detected by regenerating the image from its predicted labels and learning a dissimilarity metric between that reconstruction and the original input.

desk verdict An honest write-up of a plausible idea whose own experiments show it does not beat a softmax baseline; useful as a negative result, not as a positive claim. read the letter →

arxiv 1909.00676 v1 pith:BDQSUTZ5 submitted 2019-09-02 cs.CV

classification cs.CV
keywords semanticsegmentationout-of-distributiondetectionconditionalGANdissimilaritymetrictripletlearningnoveltyautonomousdrivingimagereconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a semantic segmentation network's own errors can be detected from outside the network, by using a conditional generative model to redraw the image from the predicted label map and then learning what that redrawing gets wrong. The intended payoff is a wrapper that can be attached to any segmentation network to localize misclassified or out-of-distribution objects without retraining the segmenter. The authors implement the generator with a conditional GAN, train three patch-based dissimilarity detectors on triplets, and test them on out-of-distribution scenes and misclassification cases. In the reported experiments the learned detectors lag behind a simple softmax-entropy baseline and suffer from false positives caused by generator instability and brightness sensitivity. The paper's contribution is the framing—visual dissimilarity as a local error signal for segmentation—rather than a deployment-ready system.

What carries the argument

The load-bearing mechanism is a two-stage reconstruction-and-compare loop. A conditional GAN (pix2pix, in the paper's implementation) maps a semantic label map to a synthetic RGB image; the hope is that within the training domain, a correct label map gives a plausible reconstruction while an erroneous label map does not. The second stage is a patch-based dissimilarity detector whose feature extractor is the first seven layers of VGG16, chosen to keep the receptive field local, followed by one of three decision heads: 1x1 convolution with bilinear upsampling, deconvolutions, or fully connected layers. The detector is trained on triplets consisting of a real/synthetic patch pair and a hard negative sampled from a different synthetic image with sufficiently different class affiliation. The objective is $L(D) = \lambda_D \mathbb{E}_{t_i}[\log D(p_i^+)] + \mathbb{E}_{t_i}[\log(1 - D(p_i^-))]$, which the paper minimizes to make the detector assign high scores to inconsistent patch pairs.

What would settle it

Use a validation set with ground-truth labels, generate a synthetic image from those labels, and then corrupt a known region of the label map (for example, change road pixels to grass). If the learned dissimilarity detector does not score the corrupted region substantially higher than the uncorrupted region across many images, the central mechanism is not carrying the detection. The same test applied to the paper's own pipeline would isolate whether the low AUC stems from the generator, the detector, or the premise itself.

Watch

Extended reading notes

Core claim

The central claim is that a wrong semantic segmentation is visible as a visual inconsistency after the scene is regenerated from the labels. More precisely, the paper proposes passing the predicted segmentation map through a conditional GAN to synthesize an RGB image, then training a dissimilarity detector that compares local patches of the real image and the synthetic reconstruction. The detector is trained with triplets so it learns to recognize pairs that look alike versus pairs that differ, and it outputs a spatial map of suspicion rather than a single image-level score. The paper's own results show that the method can flag some out-of-distribution objects and misclassifications, but all three detector variants score lower area under the ROC curve (around 0.51–0.55) than the softmax-entropy baseline (0.73) on the out-of-distribution test, and the authors attribute the shortfall to the generator's failure to converge, its low output resolution, and the detector's sensitivity to brightness differences. The claim being argued is therefore conditional: the mechanism is sound in principle, but the generator and detector need to be substantially more reliable before the dissimilarity signal can serve as a trustworthy error detector.

Load-bearing premise

The entire pipeline rests on the assumption that a correct segmentation lets the generator redraw the image faithfully, so that a wrong segmentation stands out as a visual mismatch; in the paper's experiments this assumption did not hold because the conditional GAN failed to converge and produced low-resolution, brightness-sensitive reconstructions.

Editorial extensions

If this is right

  • Any semantic segmentation network could be wrapped with this method to obtain per-pixel error and out-of-distribution detection without modifying or retraining the segmenter itself.
  • Because the dissimilarity output is local and spatial, it can feed a downstream decision layer (for example, in autonomous driving) that needs to know where, not just whether, the segmentation is unreliable.
  • The success of the method depends mostly on the generative model: a higher-resolution, stable conditional generator would directly improve detection, since the main failure mode in the experiments is the low quality and non-convergence of the generated images.
  • The fully-connected detector head performs better on misclassification, while the deconvolution head performs better on out-of-distribution instances, suggesting that detector architecture should be chosen per task.
  • In its current form the method is not competitive with the softmax-entropy baseline on global out-of-distribution detection; the paper positions it as a framework whose components require further work rather than as a finished detector.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the method only compares an input with its reconstruction from a structured output, the same wrapper could be applied to other dense prediction tasks—depth estimation, instance segmentation, or panoptic maps—wherever a generator can redraw the scene from the prediction; the paper does not explore this.
  • The systematic brightness sensitivity reported in Section 5 suggests an inexpensive remedy the authors do not test: aligning the global illumination of the generated image to the input before computing dissimilarity, which could remove the majority of false positives without retraining.
  • A decisive comparison for the framework would use a modern, high-resolution conditional generator in place of pix2pix; if the gap to the entropy baseline persists under a near-perfect generator, the dissimilarity premise itself, rather than the implementation, is the limiting factor.
  • The recurrence of low true-positive rates suggests that combining dissimilarity scores with the segmenter's own softmax entropy, rather than replacing it, might give the best of both: entropy for global confidence and dissimilarity for localizing the failures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a wrapper for semantic segmentation error detection: given an input image, a segmentation network S predicts a semantic map, a conditional GAN (pix2pix) generates a synthetic RGB image from that map, and a learned dissimilarity detector D compares the synthetic image with the original input to localize out-of-distribution (OoD) or misclassified regions. Three detector decision architectures (resize, deconvolution, fully connected) are compared with a transfer-learning variant, the pretrained GAN discriminator, and a softmax-entropy baseline. Experiments on a 100-image subset of Mapillary Vistas (OoD plus misclassifications) and on Wilddash (misclassifications only) are reported, with the Wilddash evaluation being qualitative. The paper concludes that the method shows promise but has a high false-positive rate.

Significance. If the proposed pipeline worked as intended, it would provide a segmentation-network-agnostic way to localize semantic segmentation errors, which would be valuable for safety-critical applications such as autonomous driving. The paper also connects conditional image generation with local novelty detection, an interesting combination, and it honestly reports negative results. However, the evidence presented does not support the central claim: the best proposed method reaches an AUC of only 0.5469 on the main quantitative experiment, while a simple softmax-entropy baseline reaches 0.7256, and the authors themselves document that the cGAN did not converge and that the dissimilarity signal is dominated by brightness artifacts. These results, together with the absence of error bars and the purely qualitative misclassification experiment, mean the manuscript does not currently establish that visual dissimilarity is a useful signal for segmentation error detection.

major comments (4)
  1. [§4.1, Table 1] The quantitative results contradict the central claim. The three proposed dissimilarity detectors achieve AUC scores of 0.5469, 0.5466, and 0.5051 on the Mapillary Vistas subset, while the softmax entropy baseline achieves 0.7256. The proposed methods are therefore not only close to chance but are substantially worse than a standard baseline. Since the paper's thesis is that visual dissimilarity can detect wrongly segmented or OoD instances, this result is load-bearing and the manuscript does not provide any evidence that the proposed signal adds information over prediction entropy.
  2. [§4.2] The misclassification detection experiment on Wilddash is reported only qualitatively through example outputs. There is no AUC, F1 score, or comparison with the softmax entropy baseline for this experiment, so the claim that one architecture 'performs better' or 'shows promising results' is not supported by any quantitative evidence. A quantitative evaluation with a baseline is necessary to substantiate the method's utility for the misclassification-detection use case.
  3. [§5] The limitations stated in the Discussion invalidate the premise of the method as implemented. The paper states that the cGAN 'did not reach convergence', produces low-resolution images, and that the dissimilarity detector is so sensitive to brightness that slight illumination differences trigger detections even when semantics match, causing a very high false positive rate. This means the measured dissimilarity signal may reflect low-level reconstruction artifacts rather than semantic disagreement between the input and the predicted segmentation. The core assumption from §3.1, that a correct segmentation leads to a faithful generated image, is therefore not demonstrated, and the reported near-chance AUCs are consistent with the signal being dominated by artifacts rather than semantics.
  4. [§4.1] The evaluation uses only 100 images and reports no confidence intervals or statistical significance tests. Given that the differences among the proposed methods (e.g., 0.5469 vs. 0.5466) are far smaller than plausible sampling noise, the paper's comparative claims about detector architectures are not supported. Error bars or confidence intervals over multiple evaluation subsets are needed.
minor comments (5)
  1. [§3.2, Eq. (1)] The notation in Eq. (1) is unclear: D(p_i^+) is used both as a probability-like score and as the input to a logarithmic loss; the text should define whether D outputs a probability of dissimilarity or similarity, and how the positive/negative labels are assigned.
  2. [§3.4] The negative sampling procedure relies on semantic segmentations to determine whether two patches are 'semantically different'. This is a sensible design choice, but the threshold and the source of the semantic labels (ground truth versus predicted) should be specified precisely, since they directly affect the training signal.
  3. [§4.1] The paper states that 100 images containing 'boat' or 'snow' were extracted from Mapillary Vistas, but it does not report how many images contain each class, how many are from different cities, or how the subset was balanced. This information is needed to assess the representativeness of the evaluation.
  4. [§4.3, Figure 6] The loss-weighting experiment reports only a plot of F1 scores without numeric values or axis labels that are legible in the manuscript; exact F1 values and the corresponding thresholds should be stated in the text or table.
  5. [References] Reference [11] (Isola et al.) is missing the publication venue and year; other references are formatted inconsistently (e.g., some arXiv identifiers include years and others do not). The bibliography should be brought into a consistent style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dissimilarity detector is trained and evaluated on held-out data, and no target performance is fitted.

full rationale

The paper's derivation chain is: input image -> semantic segmentation S -> cGAN G conditioned on the segmentation -> synthetic RGB image -> learned dissimilarity detector D comparing original and synthetic images to localize OoD or misclassified regions. The detector is trained on triplets whose positive pairs are real/synthetic patches from the same location and whose negative pairs are synthetic patches from another image with sufficiently different class affiliation (Section 3.4). This training signal uses real/synthetic correspondence and semantic class differences; it does not use the held-out error masks or any target quantity that the paper later 'predicts'. The quantitative evaluation on Mapillary Vistas and Wilddash is performed on held-out data, and the reported AUC values (Table 1) come from that held-out comparison rather than from fitting. The paper's central premise — that a correct segmentation leads to a faithful reconstruction and an incorrect one to a detectable inconsistency — is indeed undermined by the paper's own Section 5 limitation statement: the cGAN 'did not reach convergence', produced low-resolution images, and brightness sensitivity caused high false positives. However, that is an empirical validity failure, not circularity: the premise is an assumption from which the method proceeds, not a quantity defined in terms of the method's output. There are no load-bearing self-citations: the cited prior works (pix2pix, AdapNet, DeepLabv3+, softmax entropy baseline) are standard external components and none is used to import the target result. No 'uniqueness theorem' from the authors' prior work is invoked, and no fitted parameter is renamed as a prediction. Therefore the derivation is self-contained and not circular.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the untested premise that reconstruction error tracks segmentation error, plus the implicit assumption that the GAN and the metric transfer across datasets. No new physical entities are introduced.

free parameters (2)
  • lambda_D loss weighting = 1 (balanced)
    In Eq. 1, lambda_D weights the positive term; Section 4.3 shows F1 varies with lambda_D, and balanced weighting was best for the tested task.
  • Negative sampling pixel-difference threshold = not reported
    Section 3.4 requires a 'certain amount of pixels' to differ in semantic class for a negative patch; the exact threshold is undisclosed and affects training.
assumptions (3)
  • ad hoc to paper Reconstruction consistency is a valid signal for segmentation correctness.
    The entire method assumes that wrong segmentation yields visual dissimilarity between input and generated image; no independent justification is given.
  • domain assumption A Cityscapes-trained cGAN produces faithful reconstructions of in-distribution images.
    The paper states the GAN 'did not converge' and produces low-resolution output, which the authors themselves note impairs performance (Section 5).
  • domain assumption The dissimilarity detector transfers across datasets.
    The detector is trained on Cityscapes-style data and tested on Mapillary and Wilddash without adaptation; the paper does not analyze domain shift.

how reviews work

0 comments
Cite this review

Pith. "Pith review of This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity." pith.science (2026). https://pith.science/paper/BDQSUTZ5

@misc{pith2026190900676,
  author       = {Pith},
  title        = {Pith review of: This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDQSUTZ5}},
  note         = {Machine review of arXiv:1909.00676}
}
read the original abstract

There has been a remarkable progress in the accuracy of semantic segmentation due to the capabilities of deep learning. Unfortunately, these methods are not able to generalize much further than the distribution of their training data and fail to handle out-of-distribution classes appropriately. This limits the applicability to autonomous or safety critical systems. We propose a novel method leveraging generative models to detect wrongly segmented or out-of-distribution instances. Conditioned on the predicted semantic segmentation, an RGB image is generated. We then learn a dissimilarity metric that compares the generated image with the original input and detects inconsistencies introduced by the semantic segmentation. We present test cases for outlier and misclassification detection and evaluate our method qualitatively and quantitatively on multiple datasets.

Figures

Figures reproduced from arXiv: 1909.00676 by the authors.

Figure 1
Figure 1. Generative capabilities of adversarial networks are lever [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the pipeline showing the input that is processed by the semantic segmentation network [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The complete network structure with the feature extractor based on [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: ROC curve for (i) Resize, (ii) Deconvolution, (iii) Fully [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Dissimilarity outputs for the Deconvolution, Resize and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: F1 score performance plot of the Fully Connected ap [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative Examples from the Error Detection experiment. The Fully Connected method seems to perform better for misclassi [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Pixel to Mask: A Survey of Out-of-Distribution Segmentation

    cs.CV 2025-08 conditional novelty 3.0 of 10

    A survey categorizing out-of-distribution segmentation methods for autonomous driving into test-time, outlier-exposure, reconstruction, and powerful-model families.

Reference graph

Works this paper leans on

30 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    A. P. Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recogni- tion, 30(7):1145–1159, 1997

  2. [2]

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully con- nected crfs. IEEE transactions on pattern analysis and ma- chine intelligence, 40(4):834–848, 2018

  3. [3]

    L.-C. Chen, Y . Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for se- mantic image segmentation. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) , pages 801– 818, 2018

  4. [4]

    Chen and V

    Q. Chen and V . Koltun. Photographic image synthesis with cascaded refinement networks. In IEEE International Con- ference on Computer Vision (ICCV), volume 1, page 3, 2017

  5. [5]

    Cordts, M

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016

  6. [6]

    Y . Gal. Uncertainty in deep learning. University of Cam- bridge, 2016

  7. [7]

    Goodfellow, J

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014

  8. [8]

    X. Han, T. Leung, Y . Jia, R. Sukthankar, and A. C. Berg. Matchnet: Unifying feature and metric learning for patch- based matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3279– 3286, 2015

Show all 30 references
  1. [9]

    Harris and M

    C. Harris and M. Stephens. A combined corner and edge detector. In Alvey vision conference, volume 15, pages 10–

  2. [10]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural net- works. arXiv preprint arXiv:1610.02136, 2016

  3. [11]

    Isola, J.-Y

    P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016

  4. [12]

    Johnson, A

    J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision , pages 694–711. Springer, 2016

  5. [13]

    Kendall, V

    A. Kendall, V . Badrinarayanan, and R. Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder- decoder architectures for scene understanding. arXiv preprint arXiv:1511.02680, 2015

  6. [14]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

  7. [15]

    K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence- calibrated classifiers for detecting out-of-distribution sam- ples. arXiv preprint arXiv:1711.09325, 2017

  8. [16]

    J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 3431–3440, 2015

  9. [17]

    D. G. Lowe. Distinctive image features from scale- invariant keypoints. International journal of computer vi- sion, 60(2):91–110, 2004

  10. [18]

    Neuhold, T

    G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 4990–4999, 2017

  11. [19]

    T. Pham, V . B. Kumar, T.-T. Do, G. Carneiro, and I. Reid. Bayesian semantic instance segmentation in open set world. In Proceedings of the European Conference on Computer Vi- sion (ECCV), pages 3–18, 2018

  12. [20]

    Pidhorskyi, R

    S. Pidhorskyi, R. Almohsen, and G. Doretto. Genera- tive probabilistic novelty detection with adversarial autoen- coders. In Advances in Neural Information Processing Sys- tems, pages 6823–6834, 2018

  13. [21]

    M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko. A review of novelty detection. Signal Processing, 99:215– 249, 2014

  14. [22]

    Richter and N

    C. Richter and N. Roy. Safe visual navigation via deep learn- ing and novelty detection. 2017

  15. [23]

    Rippel, M

    O. Rippel, M. Paluri, P. Dollar, and L. Bourdev. Metric learning with adaptive density discrimination.arXiv preprint arXiv:1511.05939, 2015

  16. [24]

    Schlegl, P

    T. Schlegl, P. Seeb¨ock, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with gen- erative adversarial networks to guide marker discovery. In International Conference on Information Processing in Med- ical Imaging, pages 146–157. Springer, 2017

  17. [25]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

  18. [26]

    Valada, J

    A. Valada, J. Vertens, A. Dhall, and W. Burgard. Adap- net: Adaptive semantic segmentation in adverse environmen- tal conditions. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 4644–4651. IEEE, 2017

  19. [27]

    H.-g. Wang, X. Li, and T. Zhang. Generative adversarial net- work based novelty detection usingminimized reconstruction error. Frontiers of Information Technology & Electronic En- gineering, 19(1):116–125, 2018

  20. [28]

    J. Wang, Y . Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y . Wu. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1386–1393, 2014

  21. [29]

    Zagoruyko and N

    S. Zagoruyko and N. Komodakis. Learning to compare im- age patches via convolutional neural networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4353–4361, 2015

  22. [30]

    Zendel, K

    O. Zendel, K. Honauer, M. Murschitz, D. Steininger, and G. Fernandez Dominguez. Wilddash-creating hazard-aware benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402–416, 2018. Figure 7. Qualitative Examples from the Error Detection experime...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.