REVIEW 4 major objections 6 minor 26 references
PanSR: An Object-Centric Mask Transformer for Panoptic Segmentation
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A mask transformer that builds proposals from object centers claims to raise panoptic quality by 3.4 points on the maritime LaRS benchmark.
desk verdict Well-engineered Mask DINO extension with a real test-set bias problem: the +3.4 PQ on LaRS needs a held-out validation before it's credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the Object-Centric Proposal module (OCP). On each feature-pyramid level it predicts an object-center map, an objectness mask, and center-and-size regression maps; non-maximum suppression turns the center map into proposal locations, and each proposal's content query comes from mask-weighted pooling of features inside an approximate object mask obtained by instance voting. Around this sit three auxiliary mechanisms: proposal-aware matching, which keeps the Hungarian one-to-one matches whose box IoU with the ground truth is at least 0.25 and adds unmatched queries with IoU above 0.80 as extra supervision; object-centric mask prediction, which zeroes thing-class mask logits outside a dilated predicted box; and mask-conditioned queries sampled from ground-truth object regions during training to mimic proposal noise. The paper attributes the LaRS gains to these four pieces in its ablations.
What would settle it
Run the Table 3 ablations on a fixed validation split of LaRS instead of the sequestered test set (or evaluate the released models on LaRS val); if OCP, proposal-aware matching, and dilated-box masks do not beat the pixel-level query-selection and Hungarian-matching baselines on that held-out data, the architectural explanation for the +3.4 PQ gain collapses.
Extended reading notes
Core claim
PanSR's central claim is that reworking three components around object-centric principles closes the small-object and instance-merging gaps of mask transformers. Concretely, the paper argues that thing-class proposals should be generated at object level from predicted center maps and per-level size ranges rather than from top-k pixel activations; that matching should be many-to-one and proposal-aware so redundant proposals for the same object are kept instead of pushed onto neighbors; and that thing masks should be decoded inside a dilated bounding box so the network learns local boundaries instead of global similarity. On the LaRS maritime benchmark the paper reports a +3.4 PQ improvement over Mask DINO with Swin-L, with the largest gains in thing-class recognition quality, and a detection-rate curve that is higher across all object sizes, especially the smallest. On Cityscapes the same settings reach 67.2 PQ, matching OneFormer, which the paper reads as evidence that the design generalizes beyond maritime scenes.
Load-bearing premise
The load-bearing premise is that selecting components and hyperparameters on the sequestered LaRS test set, as the ablations in Table 3 do, does not overfit that benchmark; if it does, the claimed +3.4 PQ reflects test-set tuning rather than the architecture's design.
Editorial extensions
If this is right
- If the reported results hold, an object-centric proposal extractor can close most of the performance gap between a ResNet-50 backbone and a Swin-L backbone on scale-diverse scenes.
- Thing-class masks constrained by dilated boxes should reduce merging of visually similar instances without requiring global instance-separation features, which the qualitative comparisons support.
- Proposal-aware many-to-one matching combined with test-time NMS is a workable alternative to pure one-to-one matching in mask transformers.
- The same architecture and hyperparameters transfer from maritime to urban scenes, reaching 67.2 PQ on Cityscapes val with no dataset-specific tuning.
- The proposed components are complementary to open-vocabulary and multi-task training, so they can be combined with those directions.
Reading between the lines
- The center-map-plus-NMS proposal idea is not specific to panoptic segmentation; the same large-object bias in top-k selection appears in detection and counting, so an OCP-style module could be tested there.
- Because the ablations are run on the sequestered LaRS test set, the individual contributions to the +3.4 PQ are not yet established on unseen data; repeating the ablation on a fixed validation split would settle that.
- Bounding-box-dilated masks may hurt on highly non-convex or articulated objects, which the paper does not test; a dataset such as COCO or ADE20K would provide a harder check.
- If the mask-constrained design holds up, a practical consequence is that decoder capacity is freed from global feature separation, which may improve sample efficiency on small annotation budgets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PanSR, an object-centric mask transformer for panoptic segmentation. The method introduces three main components: an Object-Centric Proposal (OCP) module that extracts instance queries from object-center predictions rather than pixel-level top-k selection, a proposal-aware matching scheme that allows multiple queries to match one ground-truth instance while removing low-overlap matches, and an object-centric mask prediction head that zeros mask logits outside a dilated predicted bounding box for thing classes. The authors also add mask-conditioned training queries to improve robustness to proposal noise. The central empirical claim is a +3.4 PQ improvement over Mask DINO on the LaRS test set, with state-of-the-art performance on Cityscapes val (67.2 PQ), and ablation results in Table 3 attribute the gains to the OCP module, relaxed bounding-box mask constraints, and proposal-aware matching.
Significance. If the reported results are unbiased, PanSR would be a meaningful advance for panoptic segmentation in maritime and crowded scenes, where small objects and instance merging are known failure modes. The OCP module is a plausible remedy for the scale bias of top-k query selection, and the bounding-box-constrained mask prediction directly targets the merging problem. The Cityscapes result provides some evidence that the architecture does not simply overfit LaRS. The paper also includes a clear ablation structure and will release code and models, which would help reproducibility. However, the evaluation protocol substantially weakens confidence in the headline margin, because all ablations and model-selection decisions are made on the LaRS test set rather than a held-out validation split, and no repeated runs or error bars are reported.
major comments (4)
- [§4.4 (Table 3)] The ablation study and model selection are performed on the LaRS test set. Section 4.4 states explicitly: 'We ablate the major contributions of PanSR on the LaRS test set,' even though LaRS has a validation split (Section 4.2). This means the reported +3.4 PQ gain over Mask DINO is the result of selecting components and thresholds on the test set itself. The margins of some ablations (e.g., 0.3 PQ for mask-conditioned queries, 0.9 PQ for proposal-aware matching) are exactly the scale at which selection noise can act. Please rerun the ablations on the validation split, freeze a single architecture and hyperparameter configuration, and then report the test-set result once for that configuration. Reporting standard deviations or seed sensitivity would also strengthen the claim.
- [§4.2 (Table 1)] The comparison with Mask DINO is under-specified. The text says 'We apply the same training schedule to baseline methods,' but it is not clear whether Mask DINO was retrained under PanSR's exact protocol, including the copy-paste augmentation, the same optimizer settings, and the same number of iterations, or whether published numbers were quoted. This matters because Table 1 shows PanSR with ResNet-50 matching Mask DINO with Swin-L. Please document the exact retraining protocol for every baseline, including data augmentation, compute, and any modifications to the original implementation, and make the configuration files available.
- [§3.2 (Eq. 6)] The claimed reduction in instance merging is partly guaranteed by construction. Equation (6) sets the mask probability to zero outside the dilated predicted bounding box, so spatially well-separated instances cannot produce a merged mask regardless of learned feature quality. This is a legitimate inductive bias, but the paper should frame the improvement accordingly and not present it as evidence that the network has learned better instance separation. The Table 3 comparison between 'Global' and 'Relaxed BBox' changes the output space, so part of the +2.2 PQ gain is expected from the constraint itself. Please discuss what is learned versus what is enforced, and consider reporting merging rates conditioned on predicted box overlap to quantify the effect.
- [§4.4 (Table 3), §4.2] No repeated runs, error bars, or statistical significance measures are reported anywhere. All central comparisons—including the +3.4 PQ difference in Table 1 and the individual ablations in Table 3—are single runs. On a dataset with roughly 4,000 images, PQ differences of 0.3–0.9 points can be within run-to-run variation. At minimum, report multiple seeds with mean and standard deviation for the main configurations, and ideally perform model selection on validation before touching the test set.
minor comments (6)
- [Abstract and §1] There are several typos in the abstract and introduction: 'the we rethink' should be 'we rethink', 'aleviate' should be 'alleviate', and 'adn' should be 'and'.
- [§3.1.1] The sentence 'The positional queries of can be directly obtained from the regression values at the location xi' is missing an object and should read 'The positional query of object i can be directly obtained...'.
- [§3.2] Equation (6) is introduced with 'and and ϕ(·, ϵw, ϵh) is a dilation function'—remove the duplicated 'and'.
- [§3.3.1] 'multiple queries my be extracted' should be 'may be extracted'.
- [§4.3 (Table 2) and Abstract] The abstract states PanSR reaches '67.3 PQ' on Cityscapes, while Table 2 reports 67.2 PQ. Please reconcile these numbers.
- [§1] The claim that PanSR 'outperforms all state-of-the-art methods by a large margin (+3.4% PQ)' is qualified by the Cityscapes result being 'on par' with OneFormer (67.2 vs 67.2). The phrasing could more accurately distinguish the LaRS margin from the Cityscapes tie.
Circularity Check
No significant circularity; the reported gains are empirical ablation results and benchmark scores, not derivations that reduce to their inputs.
full rationale
This is an empirical architecture paper rather than a formal derivation, so most circularity patterns do not apply. The central claims are supported by ablations in Table 3: OCP versus query selection, relaxed bounding-box mask prediction versus global masks, proposal-aware matching versus Hungarian matching, and mask-conditioned queries versus DN-DETR. Equation (6) does make the mask zero outside the dilated predicted bounding box, so some reduction in spatially separated instance merging is guaranteed by construction; however, the paper presents this as an explicit design mechanism, not as a predicted emergent property, and the reported +2.2 PQ improvement is an empirical comparison against the global-mask variant. The headline +3.4 PQ on LaRS is a benchmark result obtained through an external evaluation server, not a parameter fitted to data and then renamed as a prediction. The only self-citations are to LaRS, a public benchmark with a sequestered test set authored in part by the present authors, and to a same-group NMS reference; neither is load-bearing in a circular way, and LaRS is externally falsifiable. The main genuine risk is that Section 4.4 selects components on the LaRS test set rather than a validation split, which may inflate the reported margin; this is a selection-bias and generalization concern, not circular reasoning, so it does not raise the circularity score.
Assumptions & free parameters
free parameters (6)
- θFP (FP removal threshold) =
0.25
- θFN (FN correction threshold) =
0.80
- bbox dilation margins εw, εh =
min(0.1*w, 2), min(0.1*h, 2)
- OCP level scale ranges =
s64: [256,∞], s32: [128,512], s16: [64,256], s8: [32,128], s4: [0,64]
- Loss weights =
λobj=λreg=λcenter=5, λcls=4, λmask=λbox=5
- Number of queries =
Nst=50, Nth=250, Ndn=100
assumptions (4)
- domain assumption The LaRS test set can be repeatedly evaluated without compromising statistical validity.
- domain assumption Baseline methods are compared under fair, identical training conditions.
- domain assumption Predicted bounding boxes for thing classes are reliable enough that zeroing masks outside a dilated box does not lose true positives.
- domain assumption The center-keypoint proposal prior, center prediction plus NMS, is a sufficient proposal generator for all object scales.
Cite this review
Pith. "Pith review of PanSR: An Object-Centric Mask Transformer for Panoptic Segmentation." pith.science (2026). https://pith.science/paper/TIVYJZOX
@misc{pith2026241210589,
author = {Pith},
title = {Pith review of: PanSR: An Object-Centric Mask Transformer for Panoptic Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/TIVYJZOX}},
note = {Machine review of arXiv:2412.10589}
}
read the original abstract
Panoptic segmentation is a fundamental task in computer vision and a crucial component for perception in autonomous vehicles. Recent mask-transformer-based methods achieve impressive performance on standard benchmarks but face significant challenges with small objects, crowded scenes and scenes exhibiting a wide range of object scales. We identify several fundamental shortcomings of the current approaches: (i) the query proposal generation process is biased towards larger objects, resulting in missed smaller objects, (ii) initially well-localized queries may drift to other objects, resulting in missed detections, (iii) spatially well-separated instances may be merged into a single mask causing inconsistent and false scene interpretations. To address these issues, we rethink the individual components of the network and its supervision, and propose a novel method for panoptic segmentation PanSR. PanSR effectively mitigates instance merging, enhances small-object detection and increases performance in crowded scenes, delivering a notable +3.4 PQ improvement over state-of-the-art on the challenging LaRS benchmark, while reaching state-of-the-art performance on Cityscapes. The code and models will be publicly available at https://github.com/lojzezust/PanSR.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
COCO- Stuff: Thing and Stuff Classes in Context
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO- Stuff: Thing and Stuff Classes in Context. CVPR, pages 1209–1218, 2016. 1
work page 2016
-
[2]
End- to-End Object Detection with Transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-End Object Detection with Transformers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 12346 LNCS:213–229, 2020. 2, 8
work page 2020
-
[3]
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12475–12485, 2020. 2, 6, 7
work page 2020
-
[4]
Schwing, and Alexander Kir- illov
Bowen Cheng, Alexander G. Schwing, and Alexander Kir- illov. Per-Pixel Classification is Not All You Need for Se- mantic Segmentation. Advances in Neural Information Pro- cessing Systems, 34, 2021. 1, 2
work page 2021
-
[5]
Schwing, Alexan- der Kirillov, and Rohit Girdhar
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1280–1289, 2022. 1, 2, 3, 4, 6, 7, 8
work page 2022
-
[6]
The Cityscapes Dataset for Semantic Urban Scene Understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1, 2, 7
work page 2016
-
[7]
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoc V . Le, and Barret Zoph. Sim- ple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2917–2927, 2021. 6
work page 2021
-
[8]
Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask R-CNN. In 2017 IEEE International Confer- ence on Computer Vision (ICCV) , pages 2980–2988, 2017. 5
work page 2017
Show all 26 references
-
[9]
OneFormer: One Transformer to Rule Universal Image Segmentation
Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. arXiv, 2023. 1, 2, 6, 7, 8
2023
-
[10]
Panoptic Feature Pyramid Networks
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Doll´ar. Panoptic Feature Pyramid Networks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6399–6408, 2019. 2, 6, 7
2019
-
[11]
Panoptic segmentation
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmentation. InProceed- ings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 9396–9405. IEEE Computer Society, 2019. 2, 6
2019
-
[12]
Ni, and Lei Zhang
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, and Lei Zhang. DN-DETR: Accelerate DETR Train- ing by Introducing Query DeNoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13619–13627, 2022. 2, 5, 8
2022
-
[13]
Ni, and Heung-Yeung Shum
Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung-Yeung Shum. Mask DINO: To- wards A Unified Transformer-based Framework for Object Detection and Segmentation, 2022. 1, 2, 3, 4, 5, 6, 7, 8
2022
-
[14]
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR. InInternational Conference on Learning Representations, 2021. 2
2021
-
[15]
DA VE – A Detect-and-Verify Paradigm for Low-Shot Counting
Jer Pelhan, Alan Luke ˇziˇc, Vitjan Zavrtanik, and Matej Kris- tan. DA VE – A Detect-and-Verify Paradigm for Low-Shot Counting. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23293–23302, 2024. 5
2024
-
[16]
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5463–5474, 2020. 2, 6
2020
-
[17]
Yuille, and Liang-Chieh Chen
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan L. Yuille, and Liang-Chieh Chen. Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation. In European Conference on Computer Vision, pages 108–126,
-
[18]
Upsnet: A unified panoptic segmentation network
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2
2019
-
[19]
Open-V ocabulary Panoptic Segmentation with Text-to-Image Diffusion Mod- els, 2023
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xi- aolong Wang, and Shalini De Mello. Open-V ocabulary Panoptic Segmentation with Text-to-Image Diffusion Mod- els, 2023. 2
2023
-
[20]
Ni, and Heung-Yeung Shum
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. DINO: DETR with Improved DeNoising Anchor Boxes for End-to- End Object Detection, 2022. 2
2022
-
[21]
A Simple Framework for Open-V ocabulary Segmentation and Detec- tion, 2023
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A Simple Framework for Open-V ocabulary Segmentation and Detec- tion, 2023. 1, 2, 7, 8
2023
-
[22]
Semantic Un- derstanding of Scenes Through the ADE20K Dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic Un- derstanding of Scenes Through the ADE20K Dataset. Int J Comput Vis, 127(3):302–321, 2019. 1
2019
-
[23]
Deformable DETR: 9 Deformable Transformers for End-to-End Object Detection
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai, and Sensetime Research. Deformable DETR: 9 Deformable Transformers for End-to-End Object Detection
-
[24]
LaRS: A Diverse Panoptic Maritime Obstacle Detection Dataset and Bench- mark
Lojze ˇZust, Janez Perˇs, and Matej Kristan. LaRS: A Diverse Panoptic Maritime Obstacle Detection Dataset and Bench- mark. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 20304–20314, 2023. 1, 2, 6 10 PanSR: An Object-Centric Mask Transforme...
2023
-
[25]
OCP supervision details 6.1. Center prediction ground truth To construct the ground-truth center masks ˆCobj for super- vision of the center prediciton head (see Section 3.1.1), we utilize the following procedure. First, an object center ci = (cx, cy) is extracted from each ob...
-
[26]
OCP level size range ( d) s = 64 [256 , ∞] s = 32 [128 , 512] s = 16 [64 , 256] s = 8 [32 , 128] s = 4 [0 , 64] Table 4
Additional qualitative examples In Figures 10 and 11 we present additional qualita- tive results of PanSR and state-of-the-art methods Mask DINO [13] and OneFormer[9] on the LaRS test set and Cityscapes val set respectively. OCP level size range ( d) s = 64 [256 , ∞] s = 32 [1...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.