A Mask-RCNN Baseline for Probabilistic Object Detection

Alexander C. Berg; Phil Ammirato

REVIEW 2 major objections 4 minor 12 references

A Mask-RCNN Baseline for Probabilistic Object Detection

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that a fine-tuned Mask-RCNN, adapted with four post-processing heuristics, ranks second on the PODC probabilistic object detection leaderboard with PDQ 21.432 and the highest spatial quality.

desk verdict An honest, clearly written challenge report with a real second-place score, but the baseline claim is weakened because the final model is fine-tuned on the PODC validation set and all post-processing constants are chosen on a single validation scene. read the letter →

arxiv 1908.03621 v2 pith:JHTKAWOL submitted 2019-08-09 cs.CV

Phil Ammirato , Alexander C. Berg This is my paper · ORCID

classification cs.CV

keywords probabilisticobjectdetectionPDQprobability-basedqualityMaskR-CNNbaselinesynthetictrainingdatadomaintransferpost-processingheuristics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a standard detector trained for mean average precision (mAP) can be repurposed into a competitive probabilistic object detector under the Probability-based Detection Quality (PDQ) measure. Its submission is a Mask-RCNN fine-tuned on a mix of real MSCOCO images and synthetic House3D/SunCG renders, followed by four post-processing heuristics: dropping detections below score 0.5, setting all scores to 1.0, shrinking boxes by 10%, and attaching covariance equal to 30% of box size. On the PODC leaderboard this reaches PDQ 21.432, placing second and achieving the highest spatial quality. The authors intend it as a baseline showing how mAP-oriented behavior transfers to PDQ, and they explicitly note that the score-flattening heuristic is not in the spirit of probabilistic evaluation.

What carries the argument

The object that carries the argument is the Probability-based Detection Quality (PDQ) metric. PDQ combines label quality and spatial quality for matched detections and scales by true positives relative to false positives and false negatives: $\mathrm{PDQ} = \frac{N_{\mathrm{tp}}\, a\mathrm{PDQ}}{N_{\mathrm{fp}} + N_{\mathrm{fn}} + N_{\mathrm{tp}}}$. Each detection is a probabilistic bounding box, a box with a covariance matrix at each corner. Because the metric penalizes false positives and pixels outside the true object, the paper's post-processing is engineered around it: thresholding removes false positives, shrinking boxes avoids background pixels, covariance covers the trimmed foreground, and flattening scores to 1.0 raises label quality on true positives. This metric-driven post-processing, not a learned uncertainty model, is what produces the reported score.

What would settle it

Evaluate the final pipeline on a held-out render set from the same simulator that was never used for fine-tuning or for choosing the 0.5 threshold, 10% shrink, or 30% covariance; if PDQ drops substantially below 21.432, the leaderboard result is largely an artifact of validation-set exposure rather than a general baseline.

Watch

Extended reading notes

Core claim

Fine-tuning a Mask-RCNN that was originally trained for MSCOCO mAP, then post-processing its outputs, is enough to rank near the top of a probabilistic detection benchmark. The paper removes the mask head and the 50 non-PODC classes, fine-tunes on MSCOCO plus 173,250 synthetic images from SunCG/House3D with brightness-contrast jitter, and then applies a fixed recipe: keep detections above score 0.5, reset every kept score to 1.0, add back low-scoring pairs with high overlap, shrink each box by 10% in width and height, and attach a diagonal covariance of 30% of box width and height to each corner. The authors report that this pipeline, submitted as pammirato, is second on the leaderboard with PDQ 21.432 and has the highest average spatial quality and average overall quality of detections. The result is presented not as a probabilistic model but as a baseline that future probabilistic detectors need to beat.

Load-bearing premise

The load-bearing premise is that fine-tuning on PODC validation images and picking post-processing constants on validation data still counts as evaluating generalization to the test set; if validation and test draws come from the same simulator distribution, the reported 21.432 reflects near-test exposure rather than fully independent generalization.

Editorial extensions

If this is right

A detector trained only to maximize mAP, with no probabilistic output head, can place near the top of a probabilistic detection leaderboard after simple post-processing.
Reported PDQ scores need to be compared against this kind of post-processed baseline before a method can claim that learned uncertainty estimates help.
Setting all scores to 1.0 improves PDQ because label quality is charged only on true positives, so the original score values carry no information under this metric.
In the paper's ablation, adding training data and jitter improved mAP but lowered PDQ because false positives grew, showing the two metrics reward different behaviors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The final model was fine-tuned on PODC validation images and the post-processing constants were chosen using validation scenes, so the 21.432 figure may partly reflect tuning to the evaluation distribution rather than pure generalization; a separate held-out split would be needed to confirm the baseline transfers.
The success of shrinking boxes and adding fixed covariance suggests PDQ's spatial quality can be raised without better localization; a direct test would compare this fixed recipe against learned per-object covariances.
If the same recipe transfers to other synthetic detection benchmarks, that would indicate the heuristics exploit structural properties of PDQ rather than properties of the PODC simulator specifically.
Using the published true-positive, false-positive, and false-negative counts, one can compute an upper bound on PDQ with perfect scores and covariances, quantifying how much headroom remains for genuinely probabilistic methods.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is an honest, clearly written challenge report that does what it says. A standard Mask-RCNN plus a few post-processing tricks gets 21.432 PDQ on the PODC leaderboard, second place. The heuristics are transparent, the ablations are present, and the authors own the fact that their post-processing games the PDQ formula rather than producing meaningful uncertainty estimates. That is genuinely useful for a community still figuring out what this metric rewards.

What I like: the paper is straightforwardly honest about what it did. It admits that setting all scores to 1.0 and shrinking boxes are heuristics, not principled uncertainty modeling. The ablations in Tables 2-4 are exactly what you want to see for a challenge submission. The observation that mAP-trained detectors emit a flood of low-confidence false positives under PDQ, and that a 0.5 threshold clears most of them, is a real data point. The short PDQ derivation in Section 4.1 makes each heuristic's motivation clear.

The soft spots are real. The big one: Section 5 says the final model is fine-tuned for another 5,000 iterations on the PODC validation data. Since the test images come from the same simulator as the validation images, the reported 21.432 does not isolate the method's ability to transfer from an mAP-trained detector to a new domain. The authors never ablate that validation fine-tuning step, so the score could be substantially driven by exposure to near-test data. Second, every post-processing constant (score threshold, covariance scale, box reduction factor) is chosen on a single validation scene, PODC-val0, with no error bars. For a challenge report that is acceptable, but it limits the strength of the claim that this is a general baseline. The 'confusing objects' heuristic in Section 4.4 is described as barely helping and expensive, so that one is minor.

On balance, the central argument partially holds: yes, a standard mAP-trained detector can be adapted to PDQ with simple post-processing, but the strength of that adaptation is not independently established because of the validation fine-tuning. The paper is a useful reference point, not a new capability. It gives the PDQ community a documented baseline and a warning about how easily the metric can be gamed by rescoring.

Who should read it: anyone working on probabilistic detection evaluation, or entering a PODC-style challenge. It is a good example of metric-specific post-processing and a fair comparison point for future methods.

Recommendation: if this crosses your desk, send it to peer review. It is short, honest, and the community needs baselines. But when it is reviewed, push for two changes: an ablation that removes the validation fine-tuning step, and evaluation on more than one validation scene. Those two changes would make the baseline claim clean and the paper significantly stronger.

Referee Report

2 major / 4 minor

Summary. The paper, a challenge report for the CVPR 2019 Probabilistic Object Detection Challenge (PODC), presents a fine-tuned Mask-RCNN model with hand-designed post-processing as a baseline for probabilistic object detection. The authors start from an MSCOCO-pretrained Mask-RCNN, remove mask and non-PODC classification heads, fine-tune on MSCOCO plus synthetic SunCG/House3D data with brightness/contrast jitter, and then fine-tune further on the PODC validation data. Post-processing consists of discarding detections with score below 0.5, reassigning all remaining scores to 1.0, adding back detections that overlap strongly with other low-scoring detections, shrinking boxes by 10%, and adding a covariance matrix scaled to 20-30% of box size. The method reports second place on the PODC leaderboard with PDQ 21.432 and claims the highest spatial quality. The paper includes ablations for the score threshold, covariance setting, and box reduction factor on a single validation scene.

Significance. If the stated results are taken at face value, the paper provides a useful reference point for the PODC benchmark: it shows that a conventional mAP-trained two-stage detector can be adapted to the PDQ metric with relatively simple post-processing, achieving a non-trivial leaderboard score. The paper is transparent about its choices and provides ablation tables that document the effect of each post-processing decision. These strengths make the report a potentially valuable baseline for future work on probabilistic object detection evaluation. However, the central claim that this is a clean baseline for how mAP-trained detectors behave under PDQ is weakened by the use of PODC validation data in both model fine-tuning and hyperparameter selection, as detailed in the major comments.

major comments (2)

[Section 5] The final model is fine-tuned for an additional 5,000 iterations on the PODC validation data, and the PODC test images are generated by the same simulator as the validation images. This means the reported leaderboard score of 21.432 does not isolate the behavior of an mAP-trained detector on an unseen PDQ domain; the model has been exposed to near-test-distribution data during training. The paper does not report the score of the model without this validation-data fine-tuning step, so its contribution to the leaderboard result is never measured. This is a load-bearing issue for the claimed baseline strength: without this control, the result is a tuned system rather than an independent demonstration of transfer from mAP training to PDQ evaluation. I request an ablation that evaluates the model before the validation fine-tuning step, or an explicit repositioning of the claim.
[Section 4.2-4.6 and Tables 2-4] The post-processing constants (0.5 score threshold, reassignment of all scores to 1.0, covariance scale of 20-30% of box size, and 10% box reduction) are all selected by maximizing PDQ on a single validation scene (PODC-val0). Because the same data are used for both selecting the constants and demonstrating their improvement, the increases in PDQ shown in Tables 2-4 are partly the result of fitting to the evaluation data. With only one validation scene used, there is also no estimate of variance, so the generality of the selected constants is unknown. The paper should either tune on one scene and evaluate on a separate held-out scene, or report the variance of the chosen constants across multiple validation scenes, and should state clearly that the reported improvement is not an independent estimate of generalization.

minor comments (4)

[Abstract and Section 4] There are several typos and grammatical slips: 'felixble' should be 'flexible', 'acheived' should be 'achieved', 'heursitcs' should be 'heuristics', 'test test' is repeated, 'their is' should be 'there is', and 'true postives' should be 'true positives'. These should be corrected in a revision.
[Section 4.4] The 'confusing objects' heuristic is described as increasing performance only a small amount and 'likely not worth the expensive computation', yet it is included in the final model in Section 5. No ablation table is provided for this heuristic, so its individual contribution to the final score is not documented. Adding an ablation row for this step would improve the completeness of the analysis.
[Section 5] The sentence 'learning rate or .0005' contains a typo ('or' should be 'of'), and the fine-tuning details (e.g., whether the PODC validation data are mixed with the existing training data, and how many PODC validation images are used) are not specified. Reporting these details would help reproducibility.
[Section 6] The paper claims the highest spatial quality and average overall quality of detections, but it does not provide a comparison table of PDQ, spatial quality, and average quality for the other leaderboard entries. The claim could be verified by including the relevant leaderboard numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical baseline with validation-tuned post-processing rather than deriving a prediction from its own inputs.

full rationale

This paper makes no first-principles derivation claim whose conclusion is equivalent to its premises. Its content is explicitly empirical: fine-tune a Mask-RCNN on MSCOCO plus synthetic SunCG/House3D data, add hand-crafted post-processing heuristics, and report the resulting PODC leaderboard score. The post-processing constants (score threshold 0.5, reassigning all scores to 1.0, covariance of 20-30% of box size, and 10% box reduction) are selected on PODC-val0 and reported in Tables 2-4; the final model is additionally fine-tuned on the PODC validation data (Section 5). That is benchmark tuning and possible validation-data leakage, which weakens the score as an independent generalization measure, but it is not circularity by the standard used here: no equation is defined in terms of its own output, no fitted parameter is renamed as a prediction, and the reported leaderboard score is a measured outcome rather than a quantity derived from those constants. The paper is transparent about the heuristics and even self-critiques the score-resetting choice as 'not in the spirit of PDQ.' There are no load-bearing self-citations; all cited works (PDQ, Mask R-CNN, maskrcnn-benchmark, SunCG, House3D) are external and their use does not smuggle in the paper's conclusions. The validation-tuning concern is better categorized as an experimental-validity or generalization issue than as circular reasoning.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central recipe rests on four hand-fitted post-processing constants, a fine-tuning schedule, and several domain assumptions about the synthetic data and the evaluation code. No new physical or mathematical entities are introduced.

free parameters (6)

Detection score threshold = 0.5
Set to 0.5 because Table 2 shows the highest PDQ on PODC-val0 among thresholds 0.0 to 0.7; this threshold controls the false-positive/false-negative trade-off in Eq. 3.
Detection score values = 1.0
Section 4.3 reassigns all kept detection scores to 1.0 because label quality in Eq. 1 equals the score, so this maximizes label quality for true positives; the authors admit this is not in the spirit of PDQ.
Covariance scale = 30% of box width/height
Section 4.5 and Table 3 show covariance values around 20-30% of box dimensions maximize PDQ on validation; final model uses 30%.
Box reduction factor = 10% of width and height
Section 4.6 and Table 4 show shrinking boxes by 10% improves PDQ, while 20% hurts; this reduces background pixels with high confidence in the spatial quality term.
Fine-tuning iterations = 10,000 then 5,000
Section 5 specifies 10,000 iterations on MSCOCO+SunCG+jitter and 5,000 after adding PODC validation; these are hand-set and not ablated.
Learning rate = 0.0005
Section 5 sets learning rate to 0.0005; no ablation is reported.

assumptions (5)

domain assumption PODC validation and test data are generated by the same high-fidelity simulator, so adding the validation set to training transfers to the test set.
The paper states the test data is synthetic (Section 3) and in Section 5 trains on the PODC validation data; this transfer is assumed, not measured.
domain assumption SunCG and House3D renderings are a useful proxy for the PODC synthetic domain.
Section 3 fine-tunes on 173,250 House3D-train images and notes SunCG is less realistic than PODC; the choice relies on domain similarity.
domain assumption The PDQ evaluation code from Hall et al. [1] correctly implements the equations in Section 4.1.
All reported PDQ scores come from the challenge evaluation code; correctness is taken on trust.
domain assumption The pretrained Mask-RCNN checkpoint from [7] achieves the cited 42.2 mAP and is a sound starting point.
Section 2 adopts the third-party model without independent verification.
domain assumption PODC-val0, a single scene of the validation data, is representative enough to select thresholds and covariances that generalize to the full test set.
Sections 4.2-4.6 use PODC-val0 for all ablations; this assumes no scene-specific effects.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Mask-RCNN Baseline for Probabilistic Object Detection." pith.science (2026). https://pith.science/paper/JHTKAWOL

@misc{pith2026190803621,
  author       = {Pith},
  title        = {Pith review of: A Mask-RCNN Baseline for Probabilistic Object Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JHTKAWOL}},
  note         = {Machine review of arXiv:1908.03621}
}

read the original abstract

The Probabilistic Object Detection Challenge evaluates object detection methods using a new evaluation measure, Probability-based Detection Quality (PDQ), on a new synthetic image dataset. We present our submission to the challenge, a fine-tuned version of Mask-RCNN with some additional post-processing. Our method, submitted under username pammirato, is currently second on the leaderboard with a score of 21.432, while also achieving the highest spatial quality and average overall quality of detections. We hope this method can provide some insight into how detectors designed for mean average precision (mAP) evaluation behave under PDQ, as well as a strong baseline for future work.

Figures

Figures reproduced from arXiv: 1908.03621 by the authors.

**Figure 2.** Validation images from PODC. truth object, regardless of any other detections or objects. The spatial quality, QS, is calculated based on the ground truth’s objects segmentation mask and the probabilistic bounding box outputted by the detector. Essentially, assigning higher probabilities to pixels that belong to the object as foreground improve the score, and including any background pixels outside of the ground tr… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[1]

Probabilistic Object Detection: Definition and Evaluation

David Hall, Feras Dayoub, John Skinner, Peter Corke, Gustavo Carneiro, and Niko S \" u nderhauf. Probability-based detection quality (PDQ): A probabilistic approach to detection evaluation. CoRR , abs/1811.10800, 2018

work page Pith review arXiv 2018
[2]

Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Doll\' a r, and Ross Girshick. Mask R-CNN . In Proceedings of the International Conference on Computer Vision ( ICCV ) , 2017

work page 2017
[3]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016

2016
[4]

AI2-THOR: An Interactive 3D Environment for Visual AI

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI . arXiv , 2017

work page 2017
[5]

Girshick, Kaiming He, Bharath Hariharan, and Serge J

Tsung - Yi Lin, Piotr Doll \' a r, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. CoRR , abs/1612.03144, 2016

arXiv 2016
[6]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740--755. Springer, 2014

work page 2014
[7]

maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch

Francisco Massa and Ross Girshick. maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch . https://github.com/facebookresearch/maskrcnn-benchmark, 2018. Accessed: 2018

work page 2018
[8]

Automatic differentiation in pytorch

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

2017

Show all 12 references

[9]

Imagenet large scale visual recognition challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 2015

2015
[10]

Semantic scene completion from a single depth image

Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. Proceedings of 29th IEEE Conference on Computer Vision and Pattern Recognition , 2017

2017
[11]

Building generalizable agents with a realistic and rich 3d environment

Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209 , 2018

2018 arXiv
[12]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Probabilistic Object Detection: Definition and Evaluation

David Hall, Feras Dayoub, John Skinner, Peter Corke, Gustavo Carneiro, and Niko S \" u nderhauf. Probability-based detection quality (PDQ): A probabilistic approach to detection evaluation. CoRR , abs/1811.10800, 2018

work page Pith review arXiv 2018

[2] [2]

Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Doll\' a r, and Ross Girshick. Mask R-CNN . In Proceedings of the International Conference on Computer Vision ( ICCV ) , 2017

work page 2017

[3] [3]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016

2016

[4] [4]

AI2-THOR: An Interactive 3D Environment for Visual AI

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI . arXiv , 2017

work page 2017

[5] [5]

Girshick, Kaiming He, Bharath Hariharan, and Serge J

Tsung - Yi Lin, Piotr Doll \' a r, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. CoRR , abs/1612.03144, 2016

arXiv 2016

[6] [6]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740--755. Springer, 2014

work page 2014

[7] [7]

maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch

Francisco Massa and Ross Girshick. maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch . https://github.com/facebookresearch/maskrcnn-benchmark, 2018. Accessed: 2018

work page 2018

[8] [8]

Automatic differentiation in pytorch

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

2017