Pith. sign in

REVIEW 3 major objections 8 minor 28 references

Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes

T0 review · 3 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A background-only normalizing-flow density estimator on CNN features detects foreground objects in indoor scenes and generalizes better to novel objects than a standard FCN softmax segmenter.

desk verdict Background-only density estimation is a genuinely promising route to open-set foreground/background segmentation, and the NYU results are clean, but the headline generalization numbers are partially confounded by fitting the ensemble combiner on images from the same distribution as the test set. read the letter →

arxiv 1908.00448 v4 pith:GXNU4YSX submitted 2019-08-01 cs.CV cs.RO

Nicolas Marchal , Charlotte Moraldo , Roland Siegwart , Hermann Blum , Cesar Cadena , Abel Gawel This is my paper · ORCID
classification cs.CVcs.RO
keywords backgrounddistributionforegroundindoorobjectsdescriptorsenvironmentslearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The problem for indoor robots is that they meet objects that were never in their training data. A normal segmentation network gives confident but wrong labels for such novel objects. This paper tries a different rule: learn only what the static background looks like (walls, floor, ceiling, windows), and call everything else foreground.

The method starts with a convolutional network. It takes feature vectors from several internal layers, and keeps only those that come from image patches that are purely background. Those background feature vectors are used to train a normalizing flow, a generative model that estimates a probability density. At test time, each image patch is scored by how likely its feature vector is under the learned background density. Low-likelihood patches are marked as foreground, and the resulting low-resolution likelihood map is upsampled to the full image. Scores from several layers are combined with a small logistic regression fit on a separate set of helper images.

Experiments on the NYU indoor dataset and on a custom set of 45 diverse indoor images show that the flow-based scoring beats a standard FCN softmax network on average precision and recall, and its performance drops less when moving from training-like images to novel scenes. The method is slower than the FCN decoder but much faster and lighter than a k-nearest-neighbor density baseline. The paper is honest about limitations, such as mislabeling unusual floor textures as foreground.

Extended reading notes

Core claim

The central assertion is that a normalizing flow trained only on CNN features of background pixels can produce a binary foreground/background segmentation that handles novel, out-of-distribution objects more reliably than a classical FCN softmax segmenter. Concretely, Section IV-D reports that on the generalization set with the NYU-trained encoder, the flow ensemble raises Average Precision from 40.8% (FCN softmax) to 61.6% while also improving Average Recall from 27.0% to 38.0%, with no trade-off on the NYU test set.

Load-bearing premise

The method assumes that a simple threshold on the proportion of background pixels in each feature's receptive field correctly labels features as background, foreground, or mixed, and that discarding foreground and mixed features leaves a clean background-only training set (Section III-A3). The threshold value is never reported or varied. If this labeling is wrong, the flow learns a contaminated density and the entire pipeline degrades; the paper provides no validation that the labeling step itself is sound.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes a method for binary foreground/background segmentation in indoor scenes based on density estimation in feature space. A CNN encoder (VGG-16) extracts feature maps at multiple layers; a normalizing flow (Real NVP) is trained on features labeled as background through a receptive-field proportion heuristic. At inference, negative log-likelihoods from several layers are combined via min, max, or logistic regression to produce a likelihood map, which is upsampled and thresholded to yield a binary segmentation. The authors evaluate on the NYU test set and a newly collected generalization set of 45 images, comparing against an FCN softmax baseline and kNN density estimation. The headline result is that on the generalization set with the NYU-trained encoder, the flow ensemble improves Average Precision from 40.8% to 61.6% and Average Recall from 27.0% to 38.0%.

Significance. If the reported gains are robust and not artifacts of the evaluation protocol, the paper offers a useful application of normalizing flows to open-set segmentation, with a sensible focus on background-only training and a clearly stated safety motivation. The authors provide a new 70-image benchmark (45 test, 25 fitting) and compare against both kNN density and a classical FCN, which is valuable. The method itself is largely a reapplication of the Fishyscapes recipe [11] to indoor scenes, so the novelty is incremental, but the systematic evaluation across three encoder initializations and the discussion of limitations (planar surfaces, OoD background) are strengths. Reproducibility is weakened by an unreported feature-labeling threshold and by the confounded fitting of the ensemble combiner on the same distribution as the generalization test set.

major comments (3)
  1. [IV-D / III-B3] The reported superiority of the flow ensemble over FCN softmax on the generalization set is confounded by the fact that the logistic-regression combiner (Section III-B3, iii) is fit on 25 images from the same distribution as the 45-image test set, whereas the FCN baseline is given no equivalent adaptation. The paper does not report the flow ensemble under the min/max combination rules, nor with the combiner fit on NYU data alone, so the reader cannot separate the benefit of density-based scoring from the benefit of test-distribution adaptation. This directly affects the central claim, since the headline numbers (AP 40.8% to 61.6%, AR 27.0% to 38.0% for the NYU encoder) conflate these two factors.
  2. [III-A3] The threshold used to classify features as 'background', 'foreground', or 'mixed' from the receptive-field background proportion is never reported, and no sensitivity analysis or validation of this labeling step is provided. Because this step determines the training set for the normalizing flow, an incorrect or unstable threshold could contaminate the learned background density and bias all downstream results. The method is not reproducible without reporting this threshold and its effect on the resulting segmentation.
  3. [IV-D / Conclusion] The claim that the method 'outperforms classical softmax-based segmentation on diverse and novel images' is not supported across the three encoder choices in Table II. With the ADE20K encoder, the FCN softmax has higher Average Precision (60.5 vs 51.8) and higher Average Recall (41.7 vs 27.0) than the flow ensemble; with the ImageNet encoder, the FCN has higher Average Recall (47.6 vs 40.3). Only the NYU-encoder condition shows a clear advantage, and that condition is the one affected by the confound described in the first major comment. The conclusion should be qualified to reflect the actual scope of the empirical support.
minor comments (8)
  1. [IV-B] The in-text name 'Evringham et al.' should be 'Everingham et al.' to match reference [27].
  2. [Table III] The table layout is ambiguous: the entries such as '100-50-30-40' and '1950-770-250-370' are not explicitly keyed to layers 3-6, and the title contains 'EVALUATION OFF INFERENCE TIME' which appears to be a typo for 'EVALUATION OF INFERENCE TIME'.
  3. [III-B3] The notation `¯N(zl) = N(zl)−L (Zl)` is confusing: the subscript/superscript 'l' is used inconsistently (sometimes as a layer index, sometimes as a placeholder), and the mean and standard deviation used for normalization are not defined precisely. Please clarify the notation.
  4. [V] There is a typo in the discussion: 'normlizing' should be 'normalizing'.
  5. [IV-A] There is a typo in the dataset description: 'regresion' should be 'regression'.
  6. [Conclusion] The phrase 'Unlike any existing works' is too strong, since the density-estimation-with-ensemble recipe follows Fishyscapes [11]; the paper should acknowledge the incremental nature of the contribution.
  7. [Fig. 3 caption] The caption refers to a 'weighted average of layers 3 to 6', but the method section describes a logistic regression trained on a fitting set; please harmonize the terminology.
  8. [IV-D] Given that the generalization set contains only 45 images, the paper would benefit from reporting confidence intervals or at least per-image variability for the AP and AR metrics.
Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method depends on several hand-chosen quantities (feature-labeling threshold, NLL normalization statistics, ensemble weights fit on a target-distribution set) and on domain assumptions about indoor background homogeneity and feature separability. No new physical entities are introduced.

free parameters (5)
  • Feature-labeling threshold
    Section III-A3 assigns each feature to foreground, background, or mixed using a threshold on the proportion of background pixels in the receptive field; the threshold value is never reported and directly controls which features train the density model.
  • Logistic regression ensemble weights
    Section III-B3 and IV-A use a logistic regression fit on 25 fitting images to combine per-layer NLL scores; these weights are fitted to data from the same distribution as the 45-image test set.
  • Per-layer NLL normalization statistics (mean and std)
    Section III-B3 estimates mean and standard deviation of NLL on a validation set for each layer; these statistics are inferred from data and shift the ensemble scores.
  • Number of coupling layers in Real NVP (32, with 16 also mentioned) = 32
    Section III-B2 fixes chain length at 32; Section IV-E notes 16 would cut computation with minimal quality loss. This capacity choice affects the density estimate.
  • k for kNN density
    The kNN baseline in Section III-B1 uses an unspecified k; performance and runtime depend on it.
assumptions (4)
  • domain assumption Background of indoor scenes has low variability and can be represented by a single density in CNN feature space.
    Introduction and Section III: the method's feasibility rests on background being static and homogeneous; the paper notes OoD backgrounds (e.g., parquet floors) are misclassified as foreground.
  • domain assumption Selected CNN layers produce features that separate foreground from background for the chosen encoder and datasets.
    Section III-A and IV-C: the method requires that background features cluster in the density while foreground patches are outliers; Table I shows layers 3-6 vary in quality.
  • domain assumption NYU Depth v2 labels mapped to background (wall, floor, ceiling, window) match an empty room's true structure.
    Section III-A1 defines background by this label mapping; if the mapping were wrong, the flow would model a biased background distribution.
  • ad hoc to paper The receptive-field proportion heuristic correctly labels features as background, foreground, or mixed under the unspecified threshold.
    Section III-A3 uses this heuristic to discard non-background features before training the flow; no validation of this labeling step is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes." pith.science (2026). https://pith.science/paper/GXNU4YSX

@misc{pith2026190800448,
  author       = {Pith},
  title        = {Pith review of: Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GXNU4YSX}},
  note         = {Machine review of arXiv:1908.00448}
}
read the original abstract

Deep learning has enabled remarkable advances in scene understanding, particularly in semantic segmentation tasks. Yet, current state of the art approaches are limited to a closed set of classes, and fail when facing novel elements, also known as out of distribution (OoD) data. This is a problem as autonomous agents will inevitably come across a wide range of objects, all of which cannot be included during training. We propose a novel method to distinguish any object (foreground) from empty building structure (background) in indoor environments. We use normalizing flow to estimate the probability distribution of high-dimensional background descriptors. Foreground objects are therefore detected as areas in an image for which the descriptors are unlikely given the background distribution. As our method does not explicitly learn the representation of individual objects, its performance generalizes well outside of the training examples. Our model results in an innovative solution to reliably segment foreground from background in indoor scenes, which opens the way to a safer deployment of robots in human environments.

Figures

Figures reproduced from arXiv: 1908.00448 by the authors.

Figure 1
Figure 1. Example results of the proposed feature density-based segmentation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our method where (a) shows how to extract features [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Example of results on the NYU test set obtained by extracting features from the FCN at different layers (using the encoder trained on the NYU [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example results on the generalization set (red dotted circles were [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 15 canonical work pages

  1. [11]

    The fishyscapes benchmark: Measuring blind spots in semantic segmenta- tion,

    H. Blum, P.-E. Sarlin, J. Nieto, R. Siegwart, and C. Cadena, “The fishyscapes benchmark: Measuring blind spots in semantic segmenta- tion,” arXiv preprint arXiv:1904.03215 , 2019

    arXiv 1904

  2. [1]

    Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation,

    P. R. Florence, L. Manuelli, and R. Tedrake, “Dense object nets: Learn- ing dense visual object descriptors by and for robotic manipulation,” in Conference on Robot Learning (CoRL) , 2018

    work page 2018

  3. [2]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks,

    D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations (ICLR) , 2017

    work page 2017

  4. [3]

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks,

    K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in Advances in Neural Information Processing Systems , 2018, pp. 7167– 7177

    2018

  5. [4]

    Generative ensembles for robust anomaly detec- tion,

    H. Choi and E. Jang, “Generative ensembles for robust anomaly detec- tion,” CoRR, vol. vol. abs/1810.01392v1, 2018

    arXiv 2018

  6. [5]

    Towards open set deep networks,

    A. Bendale and T. E. Boult, “Towards open set deep networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1563–1572

    work page 2016

  7. [6]

    Safety for mobile robotic systems: A systematic mapping study from a software engineering perspective,

    D. Bozhinoski, D. Di Ruscio, I. Malavolta, P. Pelliccione, and I. Crnkovic, “Safety for mobile robotic systems: A systematic mapping study from a software engineering perspective,” Journal of Systems and Software, vol. 151, pp. 150–179, 2019

    work page 2019

  8. [7]

    Mid-fusion: Octree-based object-level multi-instance dynamic slam,

    B. Xu, W. Li, D. Tzoumanikas, M. Bloesch, A. Davison, and S. Leutenegger, “Mid-fusion: Octree-based object-level multi-instance dynamic slam,” in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 5231–5237

    work page 2019

Show all 28 references
  1. [8]

    Nice: Non-linear independent components estimation,

    L. Dinh, D. Krueger, and Y . Bengio, “Nice: Non-linear independent components estimation,” in International Conference on Learning Rep- resentations (ICLR), 2015

    2015

  2. [9]

    Density estimation using real nvp,

    L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,” in International Conference on Learning Representations (ICLR) , 2017

    2017

  3. [10]

    Glow: Generative flow with invertible 1x1 convolutions,

    D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” in Advances in Neural Information Processing Sys- tems, 2018, pp. 10 236–10 245

    2018

  4. [12]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015, pp. 3431– 3440

    2015

  5. [13]

    Learning deconvolution network for semantic segmentation,

    H. Noh, S. Hong, and B. Han, “Learning deconvolution network for semantic segmentation,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1520–1528

    2015

  6. [14]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2961–2969

    2017

  7. [15]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing systems , 2015, pp. 91–99

    2015

  8. [16]

    V olumetric instance-aware semantic mapping and 3d object discovery,

    M. Grinvald, F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto, “V olumetric instance-aware semantic mapping and 3d object discovery,” IEEE Robotics and Automation Letters , vol. 4, no. 3, pp. 3037–3044, 2019

    2019

  9. [17]

    Segnet: A deep con- volutional encoder-decoder architecture for robust semantic pixel-wise labelling,

    V . Badrinarayanan, A. Handa, and R. Cipolla, “Segnet: A deep con- volutional encoder-decoder architecture for robust semantic pixel-wise labelling,” arXiv preprint arXiv:1505.07293 , 2015

    2015 arXiv

  10. [18]

    Real-time foreground–background segmentation using codebook model,

    K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis, “Real-time foreground–background segmentation using codebook model,”Real-time imaging, vol. 11, no. 3, pp. 172–185, 2005

    2005

  11. [19]

    Efficient video object co- localization with co-saliency activated tracklets,

    K. R. Jerripothula, J. Cai, and J. Yuan, “Efficient video object co- localization with co-saliency activated tracklets,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 3, pp. 744–755, 2018

    2018

  12. [20]

    Uncertainty in deep learning,

    Y . Gal, “Uncertainty in deep learning,” Ph.D. dissertation, University of Cambridge, 2016

    2016

  13. [21]

    What uncertainties do we need in bayesian deep learning for computer vision?

    A. Kendall and Y . Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” in Advances in neural information processing systems, 2017, pp. 5574–5584

    2017

  14. [22]

    Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning,

    N. Papernot and P. McDaniel, “Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning,” arXiv preprint arXiv:1803.04765, 2018

    2018 arXiv

  15. [23]

    Indoor segmentation and support inference from rgbd images,

    N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in European Conference on Computer Vision (ECCV) . Springer, 2012, pp. 746–760

    2012

  16. [24]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations (ICLR), 2015

    2015

  17. [25]

    Scene parsing through ade20k dataset,

    B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 633– 641

    2017

  18. [26]

    Distance-based confidence score for neural network classifiers,

    A. Mandelbaum and D. Weinshall, “Distance-based confidence score for neural network classifiers,” arXiv preprint arXiv:1709.09844 , 2017

    2017 arXiv

  19. [27]

    The pascal visual object classes (voc) challenge,

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisser- man, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010

    2010

  20. [28]

    Fast k nearest neighbor search using gpu,

    V . Garcia, E. Debreuve, and M. Barlaud, “Fast k nearest neighbor search using gpu,” in 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops . IEEE, 2008, pp. 1–6

    2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.