REVIEW 3 major objections 4 minor 1 cited by
Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A two-head network can segment road scenes and flag foreign objects in one forward pass, setting a new state of the art on the WildDash benchmark.
desk verdict The method comparison is solid and worth a referee, but the WildDash SOTA claim is compromised because both submitted models were trained on WildDash val before testing on WildDash test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a dense feature extractor built from a DenseNet-169 backbone, spatial pyramid pooling, and ladder-style upsampling, topped with two heads: a C-way softmax head for the 19 Cityscapes classes and a binary head outputting inlier/outlier probability. The second head is trained with cross entropy on both inlier and outlier pixels, while the classification head sees only inliers, so negative data does not distort segmentation weights. The training procedure that makes the mechanism work resizes each ImageNet-1k-bb negative object to 5% of the inlier image and pastes it at a random location before cropping, forcing the feature extractor to learn inlier/outlier boundaries rather than image-level differences.
What would settle it
Train the two-head model on the same Vistas inliers but replace ImageNet-1k-bb negatives with negatives that share no classes with road scenes, then evaluate on WildDash negative images; if outlier detection on foreign objects stays at the reported level, the claim that the specific negative data matters would be falsified. More directly, a negative test set built from text overlays and lens artifacts, neither present in ImageNet-1k, should be detected if the pasting-based training signal transfers.
Extended reading notes
Core claim
The paper's central claim is that dense outlier detection and semantic segmentation can share a convolutional representation without materially hurting either task. On the WildDash benchmark, the two-head model (LDN BIN) and the uniform-outlier C-way model (LDN OE) reach meta mIoU 41.8 and 42.7, ahead of the previous best 39.0. The binary head localizes out-of-distribution pixels in inlier context, and the two-head model avoids the false positives at semantic borders that plague single-head approaches. Training on ImageNet-1k-bb with pasting is the load-bearing ingredient: without pasting, average precision on pasted Pascal animals drops from 46.83 to 2.94.
Load-bearing premise
The method assumes that objects from ImageNet-1k-bb, resized to 5% of the inlier image and pasted before cropping, teach the model to localize real-world outliers like ego-vehicles, forklifts, text, and digital noise, even though those classes are not in ImageNet.
Editorial extensions
If this is right
- Outlier detection can be solved as a binary task over shared features, so autonomous driving systems can run segmentation and anomaly flagging in a single forward pass.
- Training on noisy negatives works even when the negative set contains inlier classes, as long as mixed batches keep inliers dominant.
- Pasting negatives into inliers is what teaches the model to localize foreign objects; image-level negative training alone does not.
- The two-head formulation preserves segmentation accuracy and avoids false outlier detections at object borders, unlike single-head C-way and C+1-way variants.
- The domain shift of the inlier training data dominates performance: models trained on Cityscapes alone treat much of WildDash as outliers, while Vistas or Vistas plus Cityscapes generalize better.
Reading between the lines
- If the pasting recipe transfers broadly, it suggests a cheap general recipe for open-set dense prediction: any large labeled image set can be turned into local anomaly examples without new annotation.
- The success of ImageNet-1k-bb over the full ImageNet hints that negative-set purity matters more than size; tests on other negative sources would clarify how far this extends.
- A testable extension is to paste negatives at multiple scales or with richer augmentations, which the paper mentions only indirectly through its later remark on scale jittering for position bias.
- Treating the outlier head as a separate decision layer may also apply to other dense tasks, such as depth or panoptic segmentation, where a binary 'I do not know this input' map is useful.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a single-pass dense outlier detector for semantic segmentation under domain shift. It extends a DenseNet-169 ladder-style feature extractor with a separate binary head that predicts a per-pixel inlier/outlier probability, while the classification head is trained only on inlier classes; the two outputs are merged by thresholding. The model is trained on Vistas and Cityscapes as inliers and on ImageNet-1k objects (bounding-box crops) pasted into inlier training images at 5% scale as an outlier source. Extensive validation compares this two-head model with C-way multi-class with outlier exposure, C+1-way, multi-label, confidence-head, and MC-dropout variants on WildDash val, LSUN, and pasted Pascal objects. The two submissions (LDN BIN and LDN OE) are reported to achieve meta average mIoU 41.8 and 42.7 on the WildDash test benchmark versus 39.0 for the previous best, constituting a claimed state of the art.
Significance. If the central claims hold, the paper provides useful evidence that a binary outlier head trained on noisy, diverse negatives can be appended to a segmentation network without degrading segmentation, and that pasting ImageNet-1k-bb objects into inlier context is an effective training recipe for localizing foreign objects at test time. The empirical protocol has notable strengths: 50-assay AP evaluation with standard deviations, multiple inlier/outlier pairings, ablations over backbones and training-data compositions, and an independent benchmark submission. The authors also report limitations candidly, including hazard sensitivity and failure cases. The main caveat is that the benchmark comparison is compromised by training on the benchmark's own validation split without a control evaluation.
major comments (3)
- [Section 4.1, Table 1, Fig. 10] The state-of-the-art claim is not apples-to-apples with previous submissions because both submitted models were trained on WildDash val (inliers). The paper gives no control model trained without WildDash val and evaluated on the WildDash test server, so the 2.8-3.7 pp meta-mIoU gain over AHiSS ROB (39.0) could partly reflect inlier statistics of the validation split rather than the proposed open-set generalization. This concern is not speculative: Fig. 10 shows that the windshield wiper is classified as inlier precisely because WildDash val contains such images, indicating that features have absorbed validation-set content. Please report a same-architecture control trained without WildDash val on the test benchmark, or clearly restrict the SOTA claim to the model configuration that uses it.
- [Sections 4.3 and 4.6, Fig. 3] The benchmark evaluation does not exercise the mixed-context setting that motivates a dense outlier map: the paper states that WildDash test has no negative objects in positive context. Consequently, the reported SOTA numbers quantify whole-image outlier detection, while the only quantitative mixed-context results (WD-Pascal) are synthetic pasted animals. The qualitative Fig. 3 detections of an ego-vehicle, a forklift, text, and digital noise are encouraging but not a quantitative substitute. The Table 5 ablation provides internal evidence that pasted ImageNet-1k-bb objects transfer to real outlier detection, so I do not regard the pasting assumption as a blocker; the missing piece is quantitative mixed-context evaluation on real data. Please add either a quantitative evaluation on an external mixed-context benchmark (e.g., Fishyscapes) or qualify the claims so that the headlined 'semantic segmentation and outlier detection' refers to the full validation setup rather than the benchmark metric.
- [Section 4.2, Tables 2 and 4] The comparison between models relies on mIoU differences as small as 0.7 pp (two-head 47.37 vs C-way multi-class 46.69; Vistas-only 47.17 vs Vistas+Cityscapes 47.78) without error bars or multiple seeds. Since the conclusion that the two-head model preserves segmentation accuracy is partly based on these differences, report standard deviations or at least results across several seeds for the principal comparisons.
minor comments (4)
- [Section 3.3] The 5% pasting scale is a free parameter, and a sensitivity analysis over pasting scale and pasting location would clarify how robust the recipe is to this choice.
- [Table 1] The column headings in Table 1 are difficult to parse; please expand abbreviations such as 'cla' and 'cat' or add a caption explanation.
- [Appendix A] There are typos and inconsistent terms in the supplementary (e.g., 'oulier' and 'Imagnet-1k'); a careful proofread would improve readability.
- [Section 4.4] The statement that training on Cityscapes leads to very low AP scores is presented as a domain-shift effect, but the Cityscapes-only model is also trained at half resolution; a brief comment on the possible influence of resolution would be useful.
Circularity Check
No circular derivation: central claims are empirical benchmark results on held-out WildDash test against external baselines; self-citations and the disclosed WildDash-val training are caveats, not circular reductions.
full rationale
The paper's central claims are empirical: a two-head architecture with a binary outlier head trained on pasted ImageNet-1k-bb negatives obtains high AP on WildDash val/LSUN/Pascal validation and sets a new state of the art on WildDash test. The WildDash test prediction is not equivalent by construction to any training input; the comparison is against external published submissions and the test set is held out. Hyperparameters and backbone choice are tuned on validation, which is model selection rather than a fitted quantity renamed as a prediction. The authors cite their own prior work for the ladder-style DenseNet feature extractor and for discriminative out-of-distribution detection, but these citations are not load-bearing: they do not assert a uniqueness theorem or forbid alternative architectures, and the losses (Tables 7-8) are standard cross-entropy/KL objectives. The paper does disclose that final models were trained on WildDash val inliers (Section 4.1) and admits in the appendix that this influences learned features (windshield wiper being classified as inlier due to WildDash val exposure). That is a benchmark-comparability caveat -- previous submissions did not train on WildDash val -- and could inflate the SOTA margin, but it is not a circular step in the derivation because the test evaluation remains independent of the fitted labels. No equation or fitted parameter is shown to reduce to the claimed result by definition. Self-citation exists but is not load-bearing; overall circularity is negligible.
Assumptions & free parameters
free parameters (9)
- lambda_KL =
0.2
- lambda_C+1 =
0.05
- lambda_TH =
0.2
- outlier_threshold_pIP =
0.5
- ODIN_temperature =
10
- ODIN_epsilon =
0.001
- pasting_scale =
5% of inlier image
- auxiliary_loss_weight =
0.4
- classifier_loss_weight =
0.6
assumptions (5)
- standard math Pixel-wise softmax cross-entropy and KL divergence are appropriate training objectives for segmentation and outlier exposure.
- domain assumption The Cityscapes 19-class ontology adequately covers inlier pixels in road-driving images.
- domain assumption ImageNet-1k-bb with bounding boxes is a suitable noisy negative dataset for road-driving outliers.
- domain assumption Shared convolutional features can serve both segmentation and outlier detection without significant deterioration of either.
- ad hoc to paper Pasting negative objects at 5% scale into inlier images approximates real outlier objects in inlier context.
Cite this review
Pith. "Pith review of Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift." pith.science (2026). https://pith.science/paper/6T5XMHGP
@misc{pith2026190801098,
author = {Pith},
title = {Pith review of: Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift},
year = {2026},
howpublished = {\url{https://pith.science/paper/6T5XMHGP}},
note = {Machine review of arXiv:1908.01098}
}
read the original abstract
Recent success on realistic road driving datasets has increased interest in exploring robust performance in real-world applications. One of the major unsolved problems is to identify image content which can not be reliably recognized with a given inference engine. We therefore study approaches to recover a dense outlier map alongside the primary task with a single forward pass, by relying on shared convolutional features. We consider semantic segmentation as the primary task and perform extensive validation on WildDash val (inliers), LSUN val (outliers), and pasted objects from Pascal VOC 2007 (outliers). We achieve the best validation performance by training to discriminate inliers from pasted ImageNet-1k content, even though ImageNet-1k contains many road-driving pixels, and, at least nominally, fails to account for the full diversity of the visual world. The proposed two-head model performs comparably to the C-way multi-class model trained to predict uniform distribution in outliers, while outperforming several other validated approaches. We evaluate our best two models on the WildDash test dataset and set a new state of the art on the WildDash benchmark.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Neural Network Meta Classifier: Improving the Reliability of Anomaly Segmentation
Replacing the logistic regression meta classifier with a lightweight fully connected network improves anomaly segmentation accuracy on the LostAndFound benchmark, and selecting proxy out-of-distribution images with sp...
Reference graph
Works this paper leans on
-
[1]
Bengio, Y., Courville, A.C., Vincent, P.: Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)
work page 2013
-
[2]
Bevandic, P., Kreso, I., Orsic, M., Segvic, S.: Discriminative out-of-distribution detection for semantic segmentation. CoRR abs/1808.07703 (2018)
arXiv 2018
-
[3]
Blum, H., Sarlin, P., Nieto, J.I., Siegwart, R., Cadena, C.: The Fishyscapes bench- mark: Measuring blind spots in semantic segmentation. CoRR abs/1904.03215
arXiv 1904
-
[4]
Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: ICLR (2019)
work page 2019
- [5]
-
[6]
In-Place Activated BatchNorm for Memory-Optimized Training of DNNs
Bul` o, S.R., Porzi, L., Kontschieder, P.: In-place activated batchnorm for memory- optimized training of dnns. CoRR, abs/1712.02616, December 5 (2017)
work page Pith review arXiv 2017
-
[7]
Machine Learning 28(1), 41–75 (Jul 1997)
Caruana, R.: Multitask learning. Machine Learning 28(1), 41–75 (Jul 1997). https://doi.org/10.1023/A:1007379606734
-
[8]
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV (2018)
work page 2018
Show all 50 references
-
[9]
In: CVPRW (2015)
Cordts, M., Omran, M., Ramos, S., Scharw¨ achter, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset. In: CVPRW (2015)
2015
-
[10]
In: CVPR
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Li, F.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255 (2009)
2009
-
[11]
CoRR abs/1802.04865 (2018)
DeVries, T., Taylor, G.W.: Learning confidence for out-of-distribution detection in neural networks. CoRR abs/1802.04865 (2018)
2018 arXiv
-
[12]
Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. ICCV pp. 2650–2658 (2015)
2015
-
[13]
Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vision (2010)
2010
-
[14]
International Journal of Robotics Research (IJRR) (2013)
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR) (2013)
2013
-
[15]
In: NIPS (2014)
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y.: Generative adversarial nets. In: NIPS (2014)
2014
-
[16]
In: ICML
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML. pp. 1321–1330 (2017)
2017
-
[17]
In: ICCV (2017)
He, K., Gkioxari, G., Dollr, P., Girshick, R.: Mask R-CNN. In: ICCV (2017)
2017
-
[18]
In: ECCV
He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. In: ECCV. pp. 346–361 (2014)
2014
-
[19]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CVPR pp. 770–778 (2016)
2016
-
[20]
In: ICLR (2017)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of- distribution examples in neural networks. In: ICLR (2017)
2017
-
[21]
In: ICLR (2019)
Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. In: ICLR (2019)
2019
-
[22]
In: CVPR (2017) 14 Bevandi´ c et al
Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017) 14 Bevandi´ c et al
2017
-
[23]
CoRR abs/1511.02680 (2015)
Kendall, A., Badrinarayanan, V., Cipolla, R.: Bayesian segnet: Model uncer- tainty in deep convolutional encoder-decoder architectures for scene understanding. CoRR abs/1511.02680 (2015)
2015 arXiv
-
[24]
Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: NIPS. pp. 5574–5584 (2017)
2017
-
[25]
In: arxiv 1805.01556 (2018)
Kong, S., Fowlkes, C.: Pixel-wise attentional gating for parsimonious pixel labeling. In: arxiv 1805.01556 (2018)
2018 arXiv
-
[26]
In: ICCV CVRSUAD 2017
Kreso, I., Krapac, J., Segvic, S.: Ladder-style densenets for semantic segmentation of large natural images. In: ICCV CVRSUAD 2017. pp. 238–245 (2017)
2017
-
[27]
CoRR abs/1905.05661 (2019)
Kreso, I., Krapac, J., Segvic, S.: Efficient ladder-style densenets for semantic seg- mentation of large images. CoRR abs/1905.05661 (2019)
2019 arXiv
-
[28]
CoRR abs/1806.03465 (2018)
Kreso, I., Orsic, M., Bevandic, P., Segvic, S.: Robust semantic segmentation with ladder-densenet models. CoRR abs/1806.03465 (2018)
2018 arXiv
-
[29]
In: NIPS
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: NIPS. pp. 6402–6413 (2017)
2017
-
[30]
In: ICLR (2018)
Lee, K., Lee, H., Lee, K., Shin, J.: Training confidence-calibrated classifiers for detecting out-of-distribution samples. In: ICLR (2018)
2018
-
[31]
In: NeurIPS (2018)
Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out- of-distribution samples and adversarial attacks. In: NeurIPS (2018)
2018
-
[32]
In: ICLR (2018)
Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: ICLR (2018)
2018
-
[33]
In: CVPR
Lin, T., Doll´ ar, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. In: CVPR. pp. 936–944 (2017)
2017
-
[34]
In: IV (2018)
Meletis, P., Dubbelman, G.: Training of convolutional networks on multiple het- erogeneous datasets for street scene semantic segmentation. In: IV (2018)
2018
-
[35]
Nalisnick, E.T., Matsukawa, A., Teh, Y.W., G¨ or¨ ur, D., Lakshminarayanan, B.: Do deep generative models know what they don’t know? In: ICLR (2019)
2019
-
[36]
In: ICCV (2017)
Neuhold, G., Ollmann, T., Bul` o, S.R., Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In: ICCV (2017)
2017
-
[37]
Ng, A.: Multimodal deep learning
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Y. Ng, A.: Multimodal deep learning. In: ICML. pp. 689–696 (2011)
2011
-
[38]
In: CVPR
Sabokrou, M., Khalooei, M., Fathy, M., Adeli, E.: Adversarially learned one-class classifier for novelty detection. In: CVPR. pp. 3379–3388 (2018)
2018
-
[39]
IEEE Trans
Scheirer, W.J., de Rezende Rocha, A., Sapkota, A., Boult, T.E.: Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1757–1772 (2013)
2013
-
[40]
Shafaei, A., Schmidt, M., Little, J.J.: Does your model know the digit 6 is not a cat? A less biased evaluation of ”outlier” detectors (2018)
2018
-
[41]
Smith, L., Gal, Y.: Understanding measures of uncertainty for adversarial example detection. In: UAI. vol. abs/1803.08533 (2018)
2018 arXiv
-
[42]
In: CVPR (June 2011)
Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR (June 2011). https://doi.org/10.1109/CVPR.2011.5995347
2011
-
[43]
In: ECCV (2018)
Vyas, A., Jammalamadaka, N., Zhu, X., Das, D., Kaul, B., Willke, T.L.: Out-of- distribution detection using an ensemble of self supervised leave-out classifiers. In: ECCV (2018)
2018
-
[44]
In: CVPR (2017)
Yu, F., Koltun, V., Funkhouser, T.: Dilated residual networks. In: CVPR (2017)
2017
-
[45]
Yu, F., Zhang, Y., Song, S., Seff, A., Xiao, J.: LSUN: construction of a large-scale image dataset using deep learning with humans in the loop (2015)
2015
-
[46]
In: CVPR (2018)
Zamir, A.R., Sax, A., Shen, W.B., Guibas, L.J., Malik, J., Savarese, S.: Taskonomy: Disentangling task transfer learning. In: CVPR (2018)
2018
-
[47]
In: ECCV (September 2018) Semantic Segmentation and Outlier Detection in Presence of Domain Shift 15
Zendel, O., Honauer, K., Murschitz, M., Steininger, D., Fernandez Dominguez, G.: Wilddash - creating hazard-aware benchmarks. In: ECCV (September 2018) Semantic Segmentation and Outlier Detection in Presence of Domain Shift 15
2018
-
[48]
International Journal of Computer Vision 125(1-3), 95–109 (2017)
Zendel, O., Murschitz, M., Humenberger, M., Herzner, W.: How good is my test data? introducing safety analysis for computer vision. International Journal of Computer Vision 125(1-3), 95–109 (2017)
2017
-
[49]
In: CVPR (2017) Appendix A Supplementary material We use this supplement to further discuss the experiments from Section 4 of the main paper
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017) Appendix A Supplementary material We use this supplement to further discuss the experiments from Section 4 of the main paper. We clarify the losses used for training the models and exp...
2017
-
[50]
trucks in the distance in the image in the third Semantic Segmentation and Outlier Detection in Presence of Domain Shift 21 Fig
as well as distant objects (cf. trucks in the distance in the image in the third Semantic Segmentation and Outlier Detection in Presence of Domain Shift 21 Fig. 6. Outlier detection with two-head models trained on different inlier datasets. All models have been trained with pas...
2007
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.