Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts

Alan Aspuru-Guzik; Sagi Eppel

REVIEW 3 major objections 5 minor 2 references

Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a generator net that guesses many imperfect segments, paired with an evaluator net that ranks them, can produce accurate panoptic segmentations and can split objects of classes never seen in training into parts.

desk verdict Novel parts-for-unseen-classes claim is real but under-proven by a lenient best-match metric; the panoptic system itself is honest and citable. read the letter →

arxiv 1908.09108 v4 pith:F4ERHPJQ submitted 2019-08-24 cs.CV cs.LG

Sagi Eppel , Alan Aspuru-Guzik This is my paper · ORCID

classification cs.CVcs.LG

keywords panopticsegmentationclass-agnosticpartsgenerator-evaluator-selectorpointernetunseenobjectclassesopen-set

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that panoptic segmentation and class-agnostic parts segmentation can be solved by a guess-and-check architecture instead of a single end-to-end model. A generator network proposes many candidate segments, most of them imperfect, and a separate evaluator network scores each candidate against how well it matches a real segment in the image; only the best-scoring segments are merged into the final map. Because each segment is generated and scored independently, the search is claimed to need exponentially fewer guesses than generating whole segmentation maps. If the claim holds, segmentation becomes a modular system of simple, interchangeable networks, and it becomes possible to split objects of classes never seen in training into meaningful parts, which the paper reports doing for the first time.

What carries the argument

The load-bearing mechanism is the separation of generation from evaluation at the level of individual segments. The Pointer net supplies variability: given an image, a pointer coordinate, and a region-of-interest mask, it outputs the segment containing that point, and different points produce different segments even inside the same object. The evaluator supplies selection: it receives the image and a candidate mask and predicts the candidate's IoU with the best real segment, allowing low-scoring proposals to be filtered out before refinement and classification. Because candidate segments are handled one at a time rather than as full segmentation maps, the scheme avoids the combinatorial explosion of guessing complete maps and keeps each module class-agnostic.

What would settle it

A decisive test would hold out an object class together with all part classes that appear on it; if parts IoU on such truly unseen objects collapses while familiar classes stay strong, the generalization claim would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a deliberately weak generator coupled with a strong evaluator is enough for competitive segmentation. The generator is a Pointer net that takes an image, a pointer point, and a region-of-interest mask and returns the mask of the segment containing that point; choosing different points yields a wide spread of candidate segments, including many wrong ones. The evaluator, a small convolutional net, predicts the intersection-over-union between each candidate and the closest real segment, so the system can discard bad guesses and stitch good ones into a panoptic map. The full system obtains a panoptic quality of 33.7 on the COCO panoptic test set, on par with basic semantic-segmentation and Mask R-CNN baselines, and for parts segmentation it achieves an IoU of 59 on unfamiliar classes in ADE20K and 48 in Pascal Parts. The paper also reports that removing the evaluator drops panoptic quality by about nine points, while replacing it with ground-truth IoU raises it by about six points, which it reads as evidence that the selection step is doing the real work.

Load-bearing premise

The unfamiliar-class result assumes that withholding object classes from training is a fair test of generalization, even though parts belonging to those classes still appear in training images, so the model may be recombining familiar part shapes rather than decomposing genuinely novel objects.

Editorial extensions

If this is right

A segmentation system can be assembled from independently trained modules, and each module can be replaced or reused without retraining the whole pipeline.
The generator does not need high average segment accuracy; it only needs to produce a wide spread of candidates that the evaluator can rank reliably.
Withholding the evaluator hurts object instances more than stuff regions, suggesting that segment selection is especially important for instance-level recognition.
Classification error is the largest single bottleneck in the pipeline, since substituting ground-truth classes raises PQ by 12.7 points.
The same generator-evaluator machinery transfers directly to parts segmentation and can propose parts for object classes that were absent from training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The unseen-class parts result probably measures compositional generalization over familiar part geometry rather than discovery of genuinely novel part types, because the paper itself notes that parts of held-out classes, such as airplane windows, appear on training classes such as buildings.
A stricter evaluation would hold out entire part families together with their object classes; such a test would tell whether the system can invent new decompositions or only recombine known ones.
The modularity argument based on guessing letter strings is an analogy rather than a bound on the actual sequential merging procedure; the real search cost could be measured by counting how many proposals the Pointer net must produce before the selected segments cover a held-out object.
The same generate-and-rank pattern could be applied to other structured prediction problems where full-solution evaluation is expensive but per-component scores are available, such as scene graph construction or document layout recovery.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper. The genuinely new piece is the claim that a pointer-net generator with an evaluator can split objects of classes never seen in training into parts. That specific capability is not in the prior work they cite. The full four-module system is also new as an assembly, even though each module is known.

What the paper does well: it releases code, it evaluates panoptic segmentation with PQ on COCO in a standard way, and it runs honest ablations. The no-evaluator drop (24.5 vs 33.7 PQ) and perfect-evaluator gain are informative; they show the generator/evaluator idea is doing real work, not just decorative. The comparison to basic Mask R-CNN + semantic segmentation baselines is fair.

The soft spot is exactly the one the stress-test flags. The parts evaluation matches each GT part mask to the most similar predicted mask, then reports IOU/precision/recall. That is a proposal-retrieval metric, not a metric on the final assembled segmentation. False part candidates that pass the 50% overlap filter never get penalized unless they are the best match for some GT part. So Table 1's IOU of 59 for unfamiliar classes is not evidence that the final output is a clean decomposition. Excluding parts under 100 pixels or 1% of the object biases toward large, easy parts. And the authors' own admission that parts of unseen classes appear in training (window on buildings vs airplanes) means the unfamiliar-object claim is really an interpolation over familiar part geometry. Those are real limitations, not manufactured ones.

The Section 3.3 exponential argument is an analogy, not a proof; the actual procedure merges overlapping segments sequentially, so the letter-string bound doesn't directly apply. I'd treat it as motivation, not as a complexity guarantee.

Citation pattern is fine. The self-citations to the pointer net and ROI classifier are to independently published pieces; no circularity.

Verdict: worth engaging. The panoptic results stand on standard metrics, and the class-agnostic parts direction is worth testing with a stricter protocol. As it stands, the headline claim is under-supported, but the underlying system is real. I'd send it to peer review and ask for a one-to-one evaluation of the final parts output, error bars, and a split where neither objects nor their parts leak from training.

Referee Report

3 major / 5 minor

Summary. The paper proposes a modular generator-evaluator-selector (GES) architecture for panoptic image segmentation and for class-agnostic parts segmentation. A pointer net generates candidate segment masks from random pointer points; an evaluator net scores each candidate by its predicted IoU with the closest real segment; selected masks are refined, classified, and stitched into a final map. For parts segmentation, the same principle is applied inside an object mask. The method reports PQ 33.7 on the COCO panoptic test set, and reports IoU 59 for parts of unfamiliar classes on ADE20K and 48 on Pascal Parts, claiming this is the first neural-net demonstration of splitting unseen object classes into parts. The paper also argues in Section 3.3 that modular segment generation requires exponentially fewer guesses than full-map generation.

Significance. If the central claims held, the paper would make two contributions: a concrete modular segmentation system with transparent ablations, and a first demonstration of class-agnostic parts segmentation for unseen object categories. The panoptic evaluation uses the standard PQ metric with honest ablations (no evaluator, perfect evaluator, no refinement, perfect classification), and the authors release code and trained models. However, the parts-segmentation result, which is the paper's most distinctive claim, is not established by the reported evaluation. The metric used for Table 1 is a best-match proposal-level evaluation of the final output, does not penalize false-positive masks in the assembled map, and excludes small parts; the paper also concedes that parts of unseen object classes appeared in training. The exponential-modularity argument is an analogy rather than a bound on the actual merging process. The central idea is interesting and the engineering is sound, but the load-bearing parts result and the complexity claim need substantial revision.

major comments (3)

[Section 5, 'Evaluating parts segmentation', Table 1] The parts evaluation matches each ground-truth part mask to the most similar predicted mask and reports IoU/precision/recall for that best match. This does not evaluate the final assembled output. Because Section 3.2 adds masks in score order and discards only masks with more than 50% overlap, many false-positive candidate masks can remain in the final map without being penalized by the reported metric; the reported precision is precision of best-matching proposals, not of the output segmentation. The evaluation also excludes parts under 100 pixels or under 1% of the object, which biases results toward larger, easier parts. To support the claim that the system 'splits objects of unseen classes into parts,' the authors should evaluate the final output map with a one-to-one matching metric (e.g., panoptic-style matching) and report sensitivity to the size threshold or remove the threshold.
[Section 5, 'Parts segmentation', Table 1 note] The paper states that while unseen object classes did not appear in training, their parts sometimes did (e.g., window appears on both buildings and airplanes). This means the unfamiliar-class experiment measures generalization over part geometry shared with seen classes, not decomposition of entirely novel object structure. The claim of demonstrating, for the first time, splitting of unseen classes into parts should be explicitly qualified by this shared-part setting, and the metric issue above should be fixed before the claim is credible.
[Section 3.3, 'On the importance of modularity'] The exponential-speedup argument is presented by analogy to guessing a string of letters. The actual system does not independently generate and evaluate each segment in a linear chain; it generates 100 candidate masks in a current ROI, merges them sequentially by score with an overlap filter, and iterates over remaining unsegmented regions. No bound or even a precise statement about the number of guesses is given for this merging process. The claim that segmentation 'will demand exponentially more guesses' compared to full-map generation should either be formalized for the actual algorithm or be presented as an intuition, not as a proven property.

minor comments (5)

[Section 3.3, reference [33]] The letter-string argument cites reference [33], which is Girshick's Fast R-CNN paper; this citation appears unrelated to the evolutionary/letter-string claim. Please check and correct the reference.
[Section 3.3, text around 'only around tries'] The sentence describing the number of tries for independent letter generation appears to have a missing numeric value (likely due to typesetting): 'will need only around ... tries.' Please correct the equation or text.
[Section 4, training details] Typo: 'for about 1–2 million interactions' should read 'iterations'.
[Table 1 and Figure 6] The paper reports no error bars or variance over the randomly selected unseen-class split. Because the group of unseen classes is small (e.g., 4 classes for Pascal Parts), a single random split could strongly affect the numbers; please report the selection procedure and, if possible, multiple splits.
[Figure 6 and unsegmented regions] Several qualitative examples show large black (unsegmented) regions. The paper does not quantify coverage, which matters for interpreting the parts results; please report the fraction of object area left unsegmented in the final output.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the GES result is tested against external benchmarks; the only self-citations are component citations that are not load-bearing.

full rationale

The paper's central derivation is not circular. The generator/evaluator system is assembled from four independently trained networks (pointer generator, evaluator, refiner, classifier) with stated losses on COCO, ADE20K, and Pascal Parts, and the headline numbers are end-to-end evaluations on held-out sets: PQ 33.7 on COCO panoptic test and parts IOU 59 for unfamiliar classes on ADE20K. No equation or reported number is a fitted parameter renamed as a prediction: the evaluator is trained to regress IOU, and the final panoptic quality is measured externally by PQ, not by the same per-proposal IOU used in training. The self-citations to the author's prior pointer net [8] and ROI classifier [9] describe component architectures, but the present paper retrains and re-evaluates these components within its own system, so the citations are not load-bearing appeals to an unverified premise. Section 3.3's claim of 'exponentially fewer guesses' is explicitly a letter-string analogy rather than a formal bound on the actual sequential segment-merging procedure, so its fragility is a rigor concern, not a circularity. The paper also candidly states that while unseen object classes did not appear in training, 'their parts sometimes did,' and the parts evaluation matches each GT part to its most similar predicted mask rather than evaluating the final assembled map; these are external-validity and evaluation-design limitations, not cases where the output is equivalent to the input by construction. No self-definitional, fitted-input, uniqueness-importation, or ansatz-smuggling pattern is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on trained CNN components and dataset assumptions, not on fitted formulas. The main risk is the unseen-class parts experiment, which assumes shared-part leakage does not invalidate generalization.

free parameters (3)

overlap_threshold = 0.5 IOU
Hand-chosen threshold in Section 3.2: predicted masks overlapping more than 50% with already added masks are ignored.
num_random_points = 100
Number of random pointer points used per object for parts segmentation (Section 3.2).
min_part_size = >100 pixels and >1% of object mask
Evaluation filter in Section 5 that excludes small parts from the parts metrics.

assumptions (4)

domain assumption Ground-truth annotations in COCO, ADE20K and Pascal Parts are correct and used as supervision.
All modules are trained on these labels and metrics are computed against them.
domain assumption Random pointer points produce a candidate distribution that covers real segments well enough for the evaluator to select good ones.
Section 3.2 relies on 100 random points per object; no coverage guarantee is given.
ad hoc to paper The letter-string analogy in Section 3.3 transfers to the segment-merging process.
The real system merges overlapping masks sequentially and does not generate segments independently, so the exponential-speedup claim is not directly implied.
domain assumption Unseen object classes are a valid test of generalization even though their parts may appear in training classes.
Section 5 admits airplane is unseen while window appears on buildings, which weakens the unfamiliar-class claim.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts." pith.science (2026). https://pith.science/paper/F4ERHPJQ

@misc{pith2026190809108,
  author       = {Pith},
  title        = {Pith review of: Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4ERHPJQ}},
  note         = {Machine review of arXiv:1908.09108}
}

read the original abstract

In machine learning and other fields, suggesting a good solution to a problem is usually a harder task than evaluating the quality of such a solution. This asymmetry is the basis for a large number of selection oriented methods that use a generator system to guess a set of solutions and an evaluator system to rank and select the best solutions. This work examines the use of this approach to the problem of panoptic image segmentation and class agnostic parts segmentation. The generator/evaluator approach for this case consists of two independent convolutional neural nets: a generator net that suggests variety segments corresponding to objects, stuff and parts regions in the image, and an evaluator net that chooses the best segments to be merged into the segmentation map. The result is a trial and error evolutionary approach in which a generator that guesses segments with low average accuracy, but with wide variability, can still produce good results when coupled with an accurate evaluator. The generator consists of a Pointer net that receives an image and a point in the image, and predicts the region of the segment containing the point. Generating and evaluating each segment separately is essential in this case since it demands exponentially fewer guesses compared to a system that guesses and evaluates the full segmentation map in each try. The classification of the selected segments is done by an independent region-specific classification net. This allows the segmentation to be class agnostic and hence, capable of segmenting unfamiliar categories that were not part of the training set. The method was examined on the COCO Panoptic segmentation benchmark and gave results comparable to those of the basic semantic segmentation and Mask-RCNN methods. In addition, the system was used for the task of splitting objects of unseen classes (that did not appear in the training set) into parts.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

[10]

Directing Attention of Convolutional Neural Networks

Nowee, S. "Directing Attention of Convolutional Neural Networks." [11] Caesar, Holger, Jasper Uijlings, and Vittorio Ferrari. "Coco-stuff: Thing and stuff classes in context." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1209-1218. 2018

work page 2018
[12]

Evolving Deep Neural Networks

Mitchell, Melanie. An introduction to genetic algorithms . MIT press, 1998. [13] Miikkulainen, Risto, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju et al. "Evolving Deep Neural Networks." arXiv preprint arXiv:1703.00548 (2017). [14] Sun, Yanan, Bing Xue, and Mengjie Zhang. "Evolving Deep Convolutional Neural Netw...

arXiv 2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [10]

Directing Attention of Convolutional Neural Networks

Nowee, S. "Directing Attention of Convolutional Neural Networks." [11] Caesar, Holger, Jasper Uijlings, and Vittorio Ferrari. "Coco-stuff: Thing and stuff classes in context." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1209-1218. 2018

work page 2018

[2] [12]

Evolving Deep Neural Networks

Mitchell, Melanie. An introduction to genetic algorithms . MIT press, 1998. [13] Miikkulainen, Risto, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju et al. "Evolving Deep Neural Networks." arXiv preprint arXiv:1703.00548 (2017). [14] Sun, Yanan, Bing Xue, and Mengjie Zhang. "Evolving Deep Convolutional Neural Netw...

arXiv 2017