REVIEW 3 major objections 6 minor 31 references
Occlusion-aware position embedding modulation improves panoptic segmentation by 0.8 PQ with minimal overhead.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
2026-08-01 15:56 UTC pith:OHV4YWCP
load-bearing objection PEMOLA is a sensible, small-gain occlusion module whose central empirical claim is undercut by an unspecified inference-time label source; worth a serious referee, but needs revision. the 3 major comments →
Occlusion-Aware Panoptic Segmentation with Joint Position Embedding and Occlusion-Level Attention
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that position embeddings — normally content-agnostic spatial encodings — can be made occlusion-aware by a residual scaling E_pos^m = E_pos ⊙ (1 + O_a ⊗ O_l), where O_a is a Grad-CAM attention map extracted from an occlusion classifier and O_l is a learnable embedding of the image's occlusion level. The authors train a classifier on COCO-OLAC (with non-object regions blackened) to produce O_a and O_l, and insert PEMOLA only into the pixel decoder. On COCO-OLAC, this raises Mask2Former from 40.7 to 41.5 PQ and Mask DINO from 44.0 to 44.8 PQ; on Cityscapes-OLAC, Mask2Former goes from 61.5 to 62.3 PQ. The paper also contributes an occlusion-level annotation for Cityscapes (C
What carries the argument
The modulation identity in Eq. (5): E_pos^m = E_pos ⊙ (1 + O_a ⊗ O_l). O_a is the ReLU-weighted Grad-CAM map from the occlusion classifier, capturing where occlusion cues appear; O_l is a C-dimensional learnable vector selected by the occlusion label L, distributing the modulation across channels. The residual '+1' preserves the original position structure, and the outer product O_a ⊗ O_l couples spatial and channel modulation. This is applied to the position embedding that feeds the pixel decoder, not the transformer decoder.
Load-bearing premise
The reported gains assume the occlusion level L used to select the embedding is available at inference; if the paper uses ground-truth L in experiments instead of the classifier's prediction, the improvement is inflated and the method is not directly deployable.
What would settle it
Run the same segmentation models with PEMOLA feeding (a) ground-truth occlusion labels and (b) the classifier's predicted labels, holding everything else fixed; if (b) loses most of the 0.8 PQ gain, the result depends on label leakage rather than on learned occlusion modelling. A confusion-matrix breakdown of the classifier under high-occlusion images would also show which errors are most damaging.
If this is right
- PEMOLA yields consistent PQ gains on COCO-OLAC (+0.8 for Mask2Former, +0.8 for Mask DINO) and Cityscapes-OLAC (+0.8 for Mask2Former), with larger relative gains under high occlusion for Mask DINO.
- Removing either occlusion prior (attention or label embedding) reduces PQ by about 0.5–0.6, showing both components contribute.
- The modulation must be inserted in the pixel decoder; inserting it in the transformer decoder gives no improvement (40.7 vs 41.5 PQ).
- Grad-CAM smoothing adds a small but consistent improvement (41.0 to 41.5 PQ).
- The approach adds minimal computational overhead because it only rescales the existing position embedding.
Where Pith is reading between the lines
- If the occlusion label at inference is taken from the classifier's argmax rather than ground truth, the effective gain may shrink; the paper does not clarify this, and the classifier's 73–78% top-1 accuracy means a robustness check against label noise is a natural next experiment.
- Because O_a is a static map produced by a frozen classifier, PEMOLA injects a fixed spatial prior per image; a natural extension is to make the attention map dynamically refined by the segmentation model itself.
- The Cityscapes-OLAC annotations could seed other dense tasks (depth, tracking) where occlusion priors are known to matter, not just panoptic segmentation.
- The residual scaling formulation suggests a general recipe: any auxiliary low-cost signal (depth, saliency, weather) could be folded into position embeddings the same way, making the idea broader than occlusion alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PEMOLA, a module for transformer-based panoptic segmentation that modulates the pixel-decoder position embedding using two occlusion priors: a Grad-CAM occlusion-level attention map extracted from a separately trained image-level occlusion classifier, and a learnable embedding of the occlusion label. The occlusion classifier is trained on the authors' COCO-OLAC dataset with non-object regions blackened. The authors also introduce Cityscapes-OLAC, a new annotation of Cityscapes with the same occlusion-level protocol, and evaluate PEMOLA with Mask2Former and Mask DINO on both datasets, reporting consistent gains of about +0.8 PQ overall, with larger per-level gains at low occlusion. The paper claims that PEMOLA is lightweight, architecture-agnostic, and improves occlusion-aware panoptic segmentation.
Significance. If the method works as described, it offers a simple and potentially generalizable way to inject occlusion priors into transformer-based panoptic segmentation, and the Cityscapes-OLAC annotation is a useful community resource. The authors also make code and data available, which supports reproducibility. However, the validity of the central claim depends critically on the inference-time data flow: the paper never states whether the occlusion label and the blackening masks used to compute the Grad-CAM attention are predicted or come from ground truth. Given that the occlusion classifier itself has only 78.2% top-1 accuracy on the black-background validation set, this ambiguity is not a minor detail; it determines whether the reported gains are real or an artifact of label leakage. The reported improvements are also small (0.5–0.8 PQ in most conditions) and are presented without variance or significance testing, so the strength of the evidence is currently insufficient.
major comments (3)
- [Section III-C, Eq. (5), and Section IV] The paper does not specify the inference-time source of the occlusion label L and of the blackening masks used to produce the occlusion-level attention O_a. Eq. (5) uses E(L) with L∈{0,1,2}, but Section IV never states whether L is the occlusion classifier's argmax or the ground-truth label. Moreover, the classifier is trained on images with non-object regions blackened (Section III-B), so computing O_a by Grad-CAM at test time requires the same blackening; this in turn requires object/panoptic masks. If L or these masks come from ground-truth annotations, the gains in Tables III–V are inflated by oracle information and the method is not deployable. The manuscript must clearly describe the inference-time pipeline and, if the cues are predicted, report experiments using predicted labels and predicted/estimated masks with an error analysis.
- [Tables III, IV, and V] The central claim of 'consistent improvement' rests on small differences (e.g., +0.8 PQ on both datasets in Tables IV and V, 0.5–0.8 PQ drops in ablations in Table VI) with no reported variance, number of runs, or significance tests. The per-level gains in Table III are larger but also lack confidence intervals. Given the classifier accuracy is only 78.2%, label prediction errors are frequent enough that the gains may shrink or disappear under a realistic inference pipeline. The authors should provide multiple-seed results with standard deviations and, at minimum, a paired comparison or significance test over the validation set to support the claim.
- [Section III-B and Cityscapes-OLAC] The blackening preprocessing is described only as 'blackening non-object regions' without specifying what masks are used (ground-truth panoptic masks? instance masks? detector outputs?) and how the same operation is applied at inference. This is essential for evaluating both the classifier's usefulness and the method's practical applicability. In addition, Cityscapes-OLAC is heavily class-imbalanced (e.g., 428 high vs. 19 low in validation), so the reported +0.8 PQ on the full set may be dominated by high-occlusion images; per-level results on Cityscapes-OLAC would clarify whether the method generalizes across levels.
minor comments (6)
- [Section III-B] Please specify the exact preprocessing for blackening non-object regions and whether the same masks are used at inference time. This is a reproducibility issue.
- [Table II] Typo in the caption: 'CLASSIFER' should be 'CLASSIFIER'.
- [Eq. (3)] The partial derivative notation should use consistent subscripts/superscripts: currently ∂Pt/∂Fk_ij mixes target index and feature indices; please clarify.
- [Figure 2] The 'No connection' label is ambiguous; a dashed outline or explicit legend would help convey that the transformer decoder receives the modulated embedding only through the pixel decoder.
- [Section IV-B] The sentence 'All models adopt ResNet-50 as the backbone, excluding the occlusion classifier' is unclear for the occlusion classifier comparison in Table I, which uses several backbones. Please rephrase.
- [Abstract / Section IV] The claim of 'minimal computational overhead' is not supported by any runtime, parameter, or FLOP measurement. Please include a quantitative comparison for the added module.
Circularity Check
No derivation-level circularity; the main issue is an unspecified inference-time label source, which would be an oracle/leakage problem rather than a circular reduction.
full rationale
The derivation chain is self-contained in the mathematical sense: Eq. (5) defines the modulated position embedding as a residual scaling of the base embedding by a learnable function of the occlusion label and a Grad-CAM attention map. Neither term is defined in terms of the panoptic segmentation output or the PQ metric; the base models (Mask2Former/Mask DINO) and the metric are external. Training an occlusion classifier on COCO-OLAC and evaluating on COCO-OLAC/Cityscapes-OLAC involves self-created benchmarks, but this is an experimental-validity concern, not a circular reduction. The most serious issue is an omitted specification: Section III-B defines L as the annotated label and Eq. (5) consumes E(L), while the experimental section never states whether L is the classifier's argmax or the ground-truth label, nor whether test-time blackening uses ground-truth masks. If ground-truth labels/masks are used, the +0.8 PQ results would be oracle artifacts; however, that would be label leakage, not a derivation that is equivalent to its inputs by construction. No equation in the paper reduces the target result to its own inputs, and no load-bearing claim rests solely on a self-citation. Hence circularity score 0.
Axiom & Free-Parameter Ledger
free parameters (3)
- learnable occlusion label embedding E(L) =
C-dimensional vectors for 3 labels, learned on COCO-OLAC
- occlusion classifier Swin-L =
trained network, 78.2% top-1 on black-background COCO-OLAC val
- Grad-CAM smoothing settings =
eigen smoothing and augmentation smoothing enabled
axioms (4)
- domain assumption COCO-OLAC image-level occlusion labels are reliable and meaningful for panoptic segmentation
- domain assumption Grad-CAM attention maps localize occlusion-relevant regions
- ad hoc to paper Blackening non-object regions preserves occlusion cues and improves classifier transfer
- domain assumption At inference, an occlusion label L is available to select E(L) without ground truth
Cite this review
Pith. "Pith review of Occlusion-Aware Panoptic Segmentation with Joint Position Embedding and Occlusion-Level Attention." pith.science (2026). https://pith.science/paper/OHV4YWCP
@misc{pith2026260718112,
author = {Pith},
title = {Pith review of: Occlusion-Aware Panoptic Segmentation with Joint Position Embedding and Occlusion-Level Attention},
year = {2026},
howpublished = {\url{https://pith.science/paper/OHV4YWCP}},
note = {Machine review of arXiv:2607.18112}
}
read the original abstract
Panoptic segmentation in complex scenes remains challenging because of occlusions, yet modern approaches often neglect occlusion modelling. In this paper, we propose Position Embedding Modulation with Occlusion Level Attention (PEMOLA), a novel occlusion-aware module that can be seamlessly integrated into transformer-based panoptic segmentation. To obtain occlusion cues, we train an occlusion classifier on the COCO-OLAC dataset. The classifier derives the occlusion-level attention, which serves as spatial guidance, while the occlusion labels are encoded into a learnable embedding to produce channel-wise weights. Through joint modulation, PEMOLA elegantly introduces the occlusion priors into the position embedding, thereby improving the occlusion modelling. We further annotate the Cityscapes dataset with occlusion levels, termed Cityscapes Occlusion Labels for All Computer Vision Tasks (Cityscapes-OLAC), following the same labelling protocol as COCO-OLAC, to evaluate the cross-dataset generalisation ability of PEMOLA. Extensive experiments on COCO-OLAC and Cityscapes-OLAC demonstrate that PEMOLA consistently improves panoptic segmentation quality while introducing minimal computational overhead. These results highlight the importance of occlusion modelling, where incorporating occlusion-level attention helps deliver robust panoptic segmentation under occlusion. Code and dataset are available at https://github.com/wenbo-wei/PEMOLA.
Figures
Reference graph
Works this paper leans on
-
[1]
Panoptic segmentation,
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Doll ´ar, “Panoptic segmentation,” inCVPR, 2019, pp. 9404–9413
2019
-
[2]
Panoptic feature pyramid networks,
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll ´ar, “Panoptic feature pyramid networks,” inCVPR, 2019, pp. 6399–6408
2019
-
[3]
Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,
Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen, “Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,” inCVPR, 2020, pp. 12475–12485
2020
-
[4]
Fully convolutional networks for panoptic segmentation,
Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia, “Fully convolutional networks for panoptic segmentation,” inCVPR, 2021, pp. 214–223
2021
-
[5]
Per-pixel classification is not all you need for semantic segmentation,
Bowen Cheng, Alex Schwing, and Alexander Kirillov, “Per-pixel classification is not all you need for semantic segmentation,”NeurIPS, vol. 34, pp. 17864–17875, 2021
2021
-
[6]
Masked-attention mask transformer for universal image segmentation,
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, “Masked-attention mask transformer for universal image segmentation,” inCVPR, 2022, pp. 1290–1299
2022
-
[7]
Mask dino: Towards a unified transformer- based framework for object detection and segmentation,
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum, “Mask dino: Towards a unified transformer- based framework for object detection and segmentation,” inCVPR, 2023, pp. 3041–3050
2023
-
[8]
You only segment once: Towards real-time panoptic segmentation,
Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, and Liujuan Cao, “You only segment once: Towards real-time panoptic segmentation,” inCVPR, 2023, pp. 17819–17829
2023
-
[9]
Maskconver: Revisiting pure convolution model for panoptic segmentation,
Abdullah Rashwan, Jiageng Zhang, Ali Taalimi, Fan Yang, Xingyi Zhou, Chaochao Yan, Liang-Chieh Chen, and Yeqing Li, “Maskconver: Revisiting pure convolution model for panoptic segmentation,” inWACV, 2024, pp. 851–861
2024
-
[10]
Shimon Ullman,High-level vision: Object recognition and visual cognition, MIT press, 2000
2000
-
[11]
Coco-olac: A benchmark for occluded panoptic segmentation and image understanding,
Wenbo Wei, Jun Wang, and Abhir Bhalerao, “Coco-olac: A benchmark for occluded panoptic segmentation and image understanding,” in ICASSP. IEEE, 2025, pp. 1–5
2025
-
[12]
Microsoft coco: Common objects in context,
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Per- ona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick, “Microsoft coco: Common objects in context,” inECCV. Springer, 2014, pp. 740– 755
2014
-
[13]
Imagenet: A large-scale hierarchical image database,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” inCVPR. Ieee, 2009, pp. 248–255
2009
-
[14]
The cityscapes dataset for semantic urban scene understanding,
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele, “The cityscapes dataset for semantic urban scene understanding,” inCVPR, 2016, pp. 3213–3223
2016
-
[15]
Amodal instance segmentation with kins dataset,
Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia, “Amodal instance segmentation with kins dataset,” inCVPR, 2019, pp. 3014– 3023
2019
-
[16]
Semantic amodal segmentation,
Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Doll ´ar, “Semantic amodal segmentation,” inCVPR, 2017, pp. 1464–1472
2017
-
[17]
Mot16: A benchmark for multi-object tracking,
Anton Milan, Laura Leal-Taix ´e, Ian Reid, Stefan Roth, and Konrad Schindler, “Mot16: A benchmark for multi-object tracking,”arXiv preprint arXiv:1603.00831, 2016
Pith/arXiv arXiv 2016
-
[18]
Attention is all you need,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,”NeurIPS, vol. 30, 2017
2017
-
[19]
Self-attention with relative position representations,
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani, “Self-attention with relative position representations,”arXiv preprint arXiv:1803.02155, 2018
Pith/arXiv arXiv 2018
-
[20]
An image is worth 16x16 words: Transformers for image recognition at scale,
Alexey Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020
Pith/arXiv arXiv 2010
-
[21]
End-to-end object detection with transformers,
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, “End-to-end object detection with transformers,” inECCV. Springer, 2020, pp. 213–229
2020
-
[22]
Dino: Detr with improved denoising anchor boxes for end-to-end object detection,
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,”arXiv preprint arXiv:2203.03605, 2022
Pith/arXiv arXiv 2022
-
[23]
Deep occlusion-aware instance segmentation with overlapping bilayers,
Lei Ke, Yu-Wing Tai, and Chi-Keung Tang, “Deep occlusion-aware instance segmentation with overlapping bilayers,” inCVPR, 2021, pp. 4019–4028
2021
-
[24]
Compositional convolutional neural networks: A robust and interpretable model for object recognition under occlusion,
Adam Kortylewski, Qing Liu, Angtian Wang, Yihong Sun, and Alan Yuille, “Compositional convolutional neural networks: A robust and interpretable model for object recognition under occlusion,”IJCV, vol. 129, pp. 736–760, 2021
2021
-
[25]
Occluded video instance segmentation: A benchmark,
Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai, “Occluded video instance segmentation: A benchmark,”IJCV, vol. 130, no. 8, pp. 2022–2039, 2022
2022
-
[26]
Mose: A new dataset for video object segmentation in complex scenes,
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai, “Mose: A new dataset for video object segmentation in complex scenes,” inICCV, 2023, pp. 20224–20234
2023
-
[27]
Mosev2: A more challenging dataset for video object segmentation in complex scenes,
Henghui Ding, Kaining Ying, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang, Philip HS Torr, and Song Bai, “Mosev2: A more challenging dataset for video object segmentation in complex scenes,” arXiv preprint arXiv:2508.05630, 2025
arXiv 2025
-
[28]
Swin transformer: Hierarchical vision transformer using shifted windows,
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inICCV, 2021, pp. 10012–10022
2021
-
[29]
Roformer: Enhanced transformer with rotary position embedding,
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu, “Roformer: Enhanced transformer with rotary position embedding,”Neurocomputing, vol. 568, pp. 127063, 2024
2024
-
[30]
Deep residual learning for image recognition,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” inCVPR, 2016, pp. 770–778
2016
-
[31]
Grad-cam: Visual explanations from deep networks via gradient-based localization,
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakr- ishna Vedantam, Devi Parikh, and Dhruv Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in ICCV, 2017, pp. 618–626
2017
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.