Instance Segmentation for Point Sets

Abhimanyu Talwar; Julien Laasri

REVIEW 4 major objections 6 minor 9 references

Instance Segmentation for Point Sets

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Instance segmentation of a 3D scene can be computed on a subsample of points and propagated to the full cloud by nearest-neighbor labeling with nearly undiminished accuracy.

desk verdict A simple subsample-and-propagate trick for SGPN's quadratic similarity matrix, with a real memory win but under-powered evidence for 'similar performance'. read the letter →

arxiv 2505.14583 v1 pith:TSKBFBAQ submitted 2025-05-20 cs.CV cs.LG

Abhimanyu Talwar , Julien Laasri This is my paper · ORCID

classification cs.CVcs.LG

keywords pointcloudinstancesegmentationSGPNsimilaritymatrixsubsamplingnearestneighborpropagationNet++meanaverageprecision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

3D instance segmentation is expensive because methods such as SGPN (Similarity Group Proposal Network) compare every point with every other point, building a similarity matrix that grows quadratically with the number of points. The paper tries to bypass that bottleneck at test time by running a pretrained SGPN on only K landmark points and then giving every other point the label of its nearest landmark in Euclidean space. Using 2048 of 4096 points in an indoor scene, it reports mean average precision only slightly below the full-point version, with the similarity matrix one quarter the size and per-block runtime down from 201 to 62 seconds. Random and grid-based landmark selection perform similarly when K is large, and random sampling is the better choice when K is small.

What carries the argument

The machinery is the N×N similarity matrix S with entries S_ij = ||F_i − F_j||_2 on per-point features; its quadratic size is both SGPN's memory bottleneck and the target of the paper's attack. The paper replaces it with two components: a landmark-selection step that picks K points either uniformly at random or as the nearest scene points to a regular grid, and a label-propagation step that builds a KDTree over the K landmarks and assigns each of the N original points the label of its nearest landmark. The resulting cost is O($K^{2}$) memory plus O(N log K) time for propagation instead of O($N^{2}$) memory, and the quantitative claim is that K=2048 is enough to preserve most of the full-point mean average precision.

What would settle it

Run the same pretrained SGPN on a held-out indoor scene, subsample to K=2048, and compare per-instance mean average precision with the full-4096 prediction; if small objects are systematically merged into nearby walls, the claim that performance stays similar is set back.

Watch

Extended reading notes

Core claim

The paper's central claim is that SGPN's quadratic memory cost can be avoided without retraining: a pretrained SGPN labels K sub-sampled landmarks, and nearest-neighbor search in the original Euclidean coordinates transfers those labels to the full point set. In the reported experiment on the 'Area 6, Office 2' scene, K=2048 of 4096 points keeps mean average precision almost at the full-point level, shrinks the similarity matrix from N×N to K×K (one quarter the entries), and reduces test time from 201 seconds to 62 seconds. At K=1024 the same pipeline starts to fail, mislabeling part of a table as wall, which marks the sparsity threshold below which small objects are lost. The two landmark-selection strategies converge as K grows, with random subsampling outperforming the grid when the landmark budget is tight.

Load-bearing premise

The load-bearing premise is that a pretrained SGPN, trained on 4096-point blocks, still gives trustworthy instance labels when given only K points; if PointNet++'s learned neighborhoods degrade under sparser, differently distributed input, the reported mAP numbers will not transfer to real scenes.

Editorial extensions

If this is right

At K=2048, test-time memory for an SGPN block drops to one quarter of the full-point similarity matrix, so scenes with too many points for a full N×N matrix become processable.
The reported per-block runtime drops from 201 seconds to 62 seconds at K=2048, making the same pretrained network practical on larger or more numerous blocks.
Random landmark selection is the preferred default at small K because it keeps mAP higher than grid selection there, while grid selection remains useful when isolated objects would be missed by random draws.
Instance quality degrades gradually with the landmark budget rather than failing outright, so the method supports a user-tunable trade-off between accuracy and memory.
The propagation scheme is agnostic to the scene size: once landmarks are labeled, any number of additional points can receive labels by nearest-neighbor lookup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would propagate labels in SGPN's learned feature space rather than in Euclidean space, since the network was trained to pull same-instance features together; this could remove most boundary errors at the same K.
Because the Euclidean nearest-neighbor rule fails on small or thin objects when K is low, an adaptive landmark budget that concentrates points where instances are dense or uncertain should beat both fixed strategies at equal K.
The same subsample-and-propagate recipe should transfer to any per-point feature extractor whose grouping stage is a quadratic similarity matrix, so the memory reduction is not specific to SGPN.
A useful follow-up measurement would separate the K-point forward pass time from the KDTree propagation time, since the reported 62-second runtime bundles both and the paper's speed claim depends on that split.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this as a course project report that found its way to arXiv. The idea is a simple workaround for SGPN's memory-hungry N×N similarity matrix: subsample to K landmarks, run the pretrained SGPN on those, and propagate labels to all points by Euclidean nearest neighbor. Both random and grid sampling are tried. The memory reduction from N² to K² is real by construction, and the runtime drop from 201s to 62s in one test room is consistent with that. The comparison of the two sampling strategies is new in this context, and the paper is honest about the failure modes: Figure 8 shows the K=1024 case mislabeling part of a table, and the grid-density caveats in Figures 4 and 5 are acknowledged rather than hidden.

The soft spots are mostly in the evaluation. Only three scenes, single runs, no error bars, no code, and the grid density is tuned per scene. That means the 'slightly worse mAP at K=2048' could be noise. The deeper concern is feature-distribution shift: the pretrained SGPN was trained on 4096-point blocks, and feeding it 2048 or 1024 points puts its ball-query neighborhoods at a density it never saw. The paper doesn't test whether the landmark predictions themselves are trustworthy; it jumps to the propagated labels. An oracle test using ground-truth labels at the landmarks would separate the sampling question from the off-distribution network question. The stress-test note is right to call this the least secure link, though it doesn't invalidate the memory argument.

The paper's contribution is a pipeline idea and a preliminary datasheet, not a validated method. Who gets value: students or practitioners who need a quick starting point for subsampling-based instance segmentation, and anyone writing a related-methods section on efficient SGPN variants. I would not cite it as authority. For peer review, it deserves a serious referee if the venue is a workshop or short-paper track — the idea is coherent and testable, but the evidence is far too thin for a main conference.

Referee Report

4 major / 6 minor

Summary. The paper proposes two sub-sampling strategies, random and grid-based, to reduce the memory and time cost of SGPN-based instance segmentation on 3D point sets. Instead of running SGPN on all N points in a block, the method selects K landmark points, obtains instance labels for those K points from a pretrained SGPN, and propagates the labels to the remaining points via Euclidean nearest-neighbor search. Experiments on three scenes from the Stanford Indoor 3D Dataset report mAP values for K = 1024, 2048, and 4096, showing that K = 2048 yields mAP only slightly lower than the full 4096-point setting while reducing the similarity matrix to one quarter of its original size and cutting computation time from 201 s to 62 s on one office scene. The paper concludes that random sub-sampling is generally preferable, while grid-based sub-sampling may help when random sampling misses small objects.

Significance. If the central claim is validated, the paper offers a simple and practical post-hoc acceleration for SGPN that requires no retraining and gives an analytic quadratic reduction in the similarity-matrix memory footprint. The algorithms are clearly specified, the evaluation uses an external pretrained SGPN and a standard public dataset, and the reported mAP numbers are not circular in the sense of being defined through the method's own fitted constants. However, the experimental evidence is preliminary: the mAP curves are single-run observations on three scenes with no error bars, no code is provided, and the grid density is tuned per scene. The method's key assumption, that a pretrained 4096-point network produces reliable labels on K-point inputs, is not directly tested. These gaps limit the strength of the 'similar performance' conclusion.

major comments (4)

[Section 3.1, Figures 6-8] The central claim of 'similar performance' rests on the assumption that a pretrained SGPN, originally trained on blocks with 4096 points, produces reliable instance labels when given only K points. This is not tested. PointNet++'s set-abstraction layers use ball queries whose local density changes when the input is sub-sampled to 2048 or 1024 points, so the features are off-distribution. Since nearest-neighbor propagation in coordinate space cannot correct labels that are wrong at the landmarks themselves, the paper needs either (a) a direct evaluation of SGPN's mAP on K-point inputs, or (b) an oracle experiment in which labels are propagated from ground-truth landmark labels to quantify the loss due to propagation alone.
[Section 4.1, Figures 6 and 7] The mAP results are single-run observations on three scenes with no error bars, no repeated trials, and no random seed specification. The difference between K = 2048 and K = 4096 could be within run-to-run variance from the random sub-sampling. The paper should report mean and standard deviation over multiple independent runs, or at least over multiple random sub-samples for each scene and value of K.
[Section 3.2, Figures 4, 5, 9, and 11] The grid density is a hyperparameter tuned per scene, as stated in Section 3.2 ('For the scenes we treated, 2048 points evenly spaced in the grid seemed to be dense enough'). No principled selection criterion or sensitivity analysis is provided. Consequently, the comparison between random and grid sub-sampling is confounded by per-scene tuning, and the claim that random sub-sampling is generally preferable is not yet established.
[Sections 3.1 and 4.1] Reproducibility is insufficient: no code is released, the SGPN implementation and pretrained model are not specified, and the runtime measurements (62 s versus 201 s) are reported without information about hardware, software versions, or the number of repetitions. Without these details, the time and memory claims cannot be independently checked.

minor comments (6)

[Abstract and Introduction] There is a typo in 'neigbhour' in the abstract; it should be 'neighbor'.
[Section 2] The word 'Voxelnet' is split as 'V oxelnet'; this should be corrected.
[Section 3.2, Algorithm 2] The algorithm is titled 'GridExtension-based sub-sampling' but the text refers to 'Grid-based' methods; please make the naming consistent.
[Figure 10 caption] The caption contains 'red-cirled'; this should be 'red-circled'.
[Section 4.1] The text says computation time is 'less than a third' of the original, while the numbers 62 s and 201 s give a ratio of about 0.31; please state this as approximate and clarify whether the reported times are wall-clock or GPU time.
[References] The Stanford 3D Indoor Spaces dataset reference [ASRZ*16] lacks the full venue and page information in the reference list; also, some references would benefit from DOIs or arXiv links for completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical comparison using an external pretrained SGPN, an external dataset, and external ground-truth labels; K and grid density are hyperparameters, not fitted constants that force the result.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. The method (Sections 3.1 and 3.2) takes a pretrained SGPN, subsamples K points from each 4096-point block, obtains instance labels for those K points from the pretrained network, and propagates the labels to all points by Euclidean nearest neighbor. The mAP values (Figures 6, 7, 9, and 11) are computed against ground-truth instance labels from the Stanford 3D Indoor Spaces Dataset, an external benchmark, and the baseline is SGPN's own full 4096-point output. No parameter is fitted to the reported mAP: K and grid density are hyperparameters chosen before evaluation, and the paper explicitly reports cases where performance degrades (e.g., K=1024 in Figure 8), showing the result is not forced by construction. The memory and time reductions follow from substituting a K×K similarity matrix for the full N×N matrix, which is a direct consequence of the construction rather than a circular prediction. The cited works (PointNet, PointNet++, SGPN) are external prior methods, and there is no load-bearing self-citation or imported uniqueness theorem. The skeptical concern about density shift is a legitimate empirical robustness risk, but it is not an instance of circularity under the stated criteria, because the paper does not define its target quantity in terms of its own parameters. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The results rest on an external pretrained model and external dataset, plus a Euclidean nearest-neighbor propagation heuristic and hand-chosen hyperparameters K and grid density.

free parameters (3)

K (number of landmarks) = 1024, 2048, and 4096 tested
Central hyperparameter for both sampling methods; the trade-off between mAP and memory/time is reported as a function of K.
Grid density (number of grid points) = 2048
Selected by visual inspection to avoid clustering artifacts in the tested scenes; affects grid-based landmark selection.
nmin and nstep = 1 and 1
Speed-up parameters in Algorithm 1; set to 1 without sensitivity analysis.

assumptions (3)

domain assumption A pretrained SGPN trained on 4096-point blocks remains reliable when applied to K-point subsamples.
Section 3.1: 'we use a pre-trained SGPN network to find an Instance Segmentation Label for each of the K points.' No retraining or adaptation is described.
domain assumption Nearest-neighbor in Euclidean space is a valid proxy for same-instance membership.
Section 3.1: 'we find the point q_K(p) among the K points which is nearest in terms of Euclidean distance to p. We then set Label(p) = Label(q_K(p)).' The paper's own Figure 8 shows boundary errors from this rule.
domain assumption SGPN's block preprocessing and merging algorithm are taken as given.
Section 3: 'we'll treat already pre-processed scenes which have 4096 points in each block' and rely on SGPN's merging step without re-evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Instance Segmentation for Point Sets." pith.science (2026). https://pith.science/paper/TSKBFBAQ

@misc{pith2026250514583,
  author       = {Pith},
  title        = {Pith review of: Instance Segmentation for Point Sets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TSKBFBAQ}},
  note         = {Machine review of arXiv:2505.14583}
}

read the original abstract

Recently proposed neural network architectures like PointNet [QSMG16] and PointNet++ [QYSG17] have made it possible to apply Deep Learning to 3D point sets. The feature representations of shapes learned by these two networks enabled training classifiers for Semantic Segmentation, and more recently for Instance Segmentation via the Similarity Group Proposal Network (SGPN) [WYHN17]. One area of improvement which has been highlighted by SGPN's authors, pertains to use of memory intensive similarity matrices which occupy memory quadratic in the number of points. In this report, we attempt to tackle this issue through use of two sampling based methods, which compute Instance Segmentation on a sub-sampled Point Set, and then extrapolate labels to the complete set using the nearest neigbhour approach. While both approaches perform equally well on large sub-samples, the random-based strategy gives the most improvements in terms of speed and memory usage.

Figures

Figures reproduced from arXiv: 2505.14583 by the authors.

**Figure 1.** Instance segmentation for different numbers of randomly or grid-based selected landmarks. K is the number of sub-sampled landmark points. Abstract Recently proposed neural network architectures like PointNet [QSMG16] and PointNet++ [QYSG17] have made it possible to apply Deep Learning to 3D point sets. The feature representations of shapes learned by these two networks enabled training classifiers for Semantic Segme… view at source ↗

**Figure 2.** Overlapping Blocks shown for one of the scenes (This is top view of the scene). based approaches to resolve this issue - the general idea is to compute Instance Segmentation on a subset of points of a scene, and then propagate their predicted labels to all points. Block Processing: The way SGPN works is that it divides a scene into 1m × 1m cuboidal blocks with overlaps of stride 0.5m (the height of each block is th… view at source ↗

**Figure 3.** Random sub-sampling method on the left misses an entire object on the left as it focuses on high density parts of the scene whereas these points would be captured by the grid-based sub-sampling method as shown on the right. on average taken O(logK) time. Therefore if we do N searches, it should take on average O(N logK) time. 3.2. Grid-based Sub-sampling Random sub-sampling is just one way of choosing landmark poin… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Two density configurations of the grid with the corresponding chosen landmarks for the same scene. On the left, the n-neighbours create small clustering patterns due to the small density of the grid in comparison with the right configuration [PITH_FULL_IMAGE:figures…

**Figure 5.** Figure 5: Finding K = 1024 landmarks for the same scene block using algorithm 1 on the left and algorithm 2 on the right. A grid size of 1024 was sufficient to produce the results on the left whereas 4096 points were needed for the same-quality result on the right. 4. Results […

**Figure 7.** Figure 7: Random Sub-sampling: Mean Average Precision and Computation Time for the scene "Area 6, Office 2" from the Stanford Indoor 3D Dataset [ASRZ∗ 16]. points inside the table (the blue area inside the red oval). However, using a sub-sample of size 1024, some of the points …

**Figure 8.** Figure 8: Predicted Instance Segmentation using the full Point Set (4096 points) and a Random Sub-sample of 1024 points for the scene "Area 6, Office 2" from the Stanford Indoor 3D Dataset [ASRZ∗ 16]. The red oval area shows some deficiencies which crop in due to sub-sampling […

**Figure 9.** Figure 9: Grid Sub-sampling: Mean Average Precision and Computation Time for the scene "Area 6, Office 2" from the Stanford Indoor 3D Dataset [ASRZ∗ 16]. clearly see the before and after difference in the bottom row of figure 1. One thing that we can notice on figure 3 is that …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

[1]

Zamir A., Jiang H., Brilakis I., Fischer M., Savarese S

Armeni I., Sener O., R. Zamir A., Jiang H., Brilakis I., Fischer M., Savarese S. : 3d semantic parsing of large-scale indoor spaces. pp. 1534--1543. http://dx.doi.org/10.1109/CVPR.2016.170 doi:10.1109/CVPR.2016.170

doi:10.1109/cvpr.2016.170 2016
[2]

Garcia - Garcia A., Orts - Escolano S., Oprea S., Villena - Martinez V., Rodr \' guez J. G. : A review on deep learning techniques applied to semantic segmentation. CoRR abs/1704.06857 (2017). URL: http://arxiv.org/abs/1704.06857, http://arxiv.org/abs/1704.06857 arXiv:1704.06857

arXiv 2017
[3]

: The handbook of brain theory and neural networks

LeCun Y., Bengio Y. : The handbook of brain theory and neural networks. MIT Press, Cambridge, MA, USA, 1998, ch. Convolutional Networks for Images, Speech, and Time Series, pp. 255--258. URL: http://dl.acm.org/citation.cfm?id=303568.303704

work page arXiv 1998
[4]

: Megadepth: Learning single-view depth prediction from internet photos

Li Z., Snavely N. : Megadepth: Learning single-view depth prediction from internet photos. CoRR abs/1804.00607 (2018). URL: http://arxiv.org/abs/1804.00607, http://arxiv.org/abs/1804.00607 arXiv:1804.00607

arXiv 2018
[5]

R., Su H., Mo K., Guibas L

Qi C. R., Su H., Mo K., Guibas L. J. : Pointnet: Deep learning on point sets for 3d classification and segmentation. CoRR abs/1612.00593 (2016). URL: http://arxiv.org/abs/1612.00593, http://arxiv.org/abs/1612.00593 arXiv:1612.00593

arXiv 2016
[6]

R., Yi L., Su H., Guibas L

Qi C. R., Yi L., Su H., Guibas L. J. : Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems 30, Guyon I., Luxburg U. V., Bengio S., Wallach H., Fergus R., Vishwanathan S., Garnett R., (Eds.). Curran Associates, Inc., 2017, pp. 5099--5108. URL: http://papers.nips.cc/paper/7095-...

work page 2017
[7]

SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation

Wang W., Yu R., Huang Q., Neumann U. : SGPN: similarity group proposal network for 3d point cloud instance segmentation. CoRR abs/1711.08588 (2017). URL: http://arxiv.org/abs/1711.08588, http://arxiv.org/abs/1711.08588 arXiv:1711.08588

work page Pith review arXiv 2017
[8]

Zhu J.-Y., Park T., Isola P., Efros A. A. : Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on (2017)

work page 2017

Show all 9 references

[9]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry.original add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.s...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Zamir A., Jiang H., Brilakis I., Fischer M., Savarese S

Armeni I., Sener O., R. Zamir A., Jiang H., Brilakis I., Fischer M., Savarese S. : 3d semantic parsing of large-scale indoor spaces. pp. 1534--1543. http://dx.doi.org/10.1109/CVPR.2016.170 doi:10.1109/CVPR.2016.170

doi:10.1109/cvpr.2016.170 2016

[2] [2]

Garcia - Garcia A., Orts - Escolano S., Oprea S., Villena - Martinez V., Rodr \' guez J. G. : A review on deep learning techniques applied to semantic segmentation. CoRR abs/1704.06857 (2017). URL: http://arxiv.org/abs/1704.06857, http://arxiv.org/abs/1704.06857 arXiv:1704.06857

arXiv 2017

[3] [3]

: The handbook of brain theory and neural networks

LeCun Y., Bengio Y. : The handbook of brain theory and neural networks. MIT Press, Cambridge, MA, USA, 1998, ch. Convolutional Networks for Images, Speech, and Time Series, pp. 255--258. URL: http://dl.acm.org/citation.cfm?id=303568.303704

work page arXiv 1998

[4] [4]

: Megadepth: Learning single-view depth prediction from internet photos

Li Z., Snavely N. : Megadepth: Learning single-view depth prediction from internet photos. CoRR abs/1804.00607 (2018). URL: http://arxiv.org/abs/1804.00607, http://arxiv.org/abs/1804.00607 arXiv:1804.00607

arXiv 2018

[5] [5]

R., Su H., Mo K., Guibas L

Qi C. R., Su H., Mo K., Guibas L. J. : Pointnet: Deep learning on point sets for 3d classification and segmentation. CoRR abs/1612.00593 (2016). URL: http://arxiv.org/abs/1612.00593, http://arxiv.org/abs/1612.00593 arXiv:1612.00593

arXiv 2016

[6] [6]

R., Yi L., Su H., Guibas L

Qi C. R., Yi L., Su H., Guibas L. J. : Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems 30, Guyon I., Luxburg U. V., Bengio S., Wallach H., Fergus R., Vishwanathan S., Garnett R., (Eds.). Curran Associates, Inc., 2017, pp. 5099--5108. URL: http://papers.nips.cc/paper/7095-...

work page 2017

[7] [7]

SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation

Wang W., Yu R., Huang Q., Neumann U. : SGPN: similarity group proposal network for 3d point cloud instance segmentation. CoRR abs/1711.08588 (2017). URL: http://arxiv.org/abs/1711.08588, http://arxiv.org/abs/1711.08588 arXiv:1711.08588

work page Pith review arXiv 2017

[8] [8]

Zhu J.-Y., Park T., Isola P., Efros A. A. : Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on (2017)

work page 2017