REVIEW 3 major objections 6 minor 45 references
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that copy-pasting objects at locations chosen by an appearance-consistency heatmap improves Mask R-CNN instance segmentation by 2.2 mAP on COCO without changing the network.
desk verdict A useful, simple augmentation trick—random jittering of instances near their original location—gives a solid +1.7 mAP on COCO, but the appearance-heatmap component adds only a small, not fully isolated gain. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery has two parts. The first is random InstaBoost: matting extracts a smooth instance patch from its mask, inpainting fills the leftover hole in the background, and the patch is pasted back under an affine transform sampled uniformly from a small neighborhood of the identity transform. The second is the appearance consistency heatmap, a probability map over paste locations built from a weighted combination of three fixed-width RGB contour rings around the object center; the heatmap is generated by measuring the appearance distance between the descriptor at the original position and descriptors at all other positions, normalizing and scaling by negative log. The heatmap is what lets the method sample semantically plausible locations far from the original position, and it is the component that distinguishes InstaBoost from random paste.
What would settle it
On a category with highly varied legal contexts, such as "person" in COCO, compare heatmap-guided InstaBoost against pure random jittering per category; if the guided version does not consistently win, the appearance proxy is not doing the semantic placement work the paper attributes to it.
Extended reading notes
Core claim
The central discovery is a training-time copy-paste augmentation that improves instance segmentation by exploiting the redundancy of background pixels. The paper shows that randomly jittering an object around its original position, with small changes in scale and rotation, contributes 1.7 mAP on COCO; adding a location probability map, called the appearance consistency heatmap, contributes another 0.5 mAP for a total of 2.2 mAP. The heatmap scores candidate paste locations by comparing the RGB values in three weighted contour rings around the object's original position with the same rings around each candidate position, then samples locations via Monte Carlo. With this guidance, R101-Mask R-CNN instance segmentation rises from 35.7 to 37.9 mAP on COCO test-dev, and Pascal VOC gains 3.3 to 3.9 mAP, all without modifying the backbone or network structure.
Load-bearing premise
The method assumes that a location whose surrounding RGB contour rings resemble the rings around an object's original position is a good place to paste that object.
Editorial extensions
If this is right
- Any Mask R-CNN-style instance segmentation model can be improved by roughly 2 mAP on COCO simply by changing the training dataloader, not the architecture.
- Training with InstaBoost postpones overfitting: accuracy keeps rising at 48 epochs where the un-augmented baseline plateaus at 24 epochs.
- Smaller datasets benefit more: on Pascal VOC, InstaBoost adds about 3.9 mAP, suggesting the method is especially useful when training data are scarce.
- Bounding-box detection also improves as a by-product, so the augmentation is not segmentation-specific.
- The method transfers to at least one other base model, Cascade R-CNN, indicating it is not tied to a single network design.
Reading between the lines
- Because the gain comes from training data, InstaBoost should combine additively with architectural improvements; one plausible next step is to treat it as a standard training recipe for any new instance segmentation model.
- The appearance heatmap could be replaced by a learned semantic compatibility score; if such a score placed objects in valid but texture-different regions, it might push beyond the 2.2 mAP ceiling reported here.
- Per-category results are not reported, so a natural test is whether gains concentrate in categories with stereotyped backgrounds (cars, animals) and shrink for categories like person, whose valid contexts vary widely.
- The same mask-plus-inpainting pipeline could be extended to video instance segmentation, where temporal coherence could supply additional valid paste locations, but the paper does not test this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes InstaBoost, an online data augmentation method for instance segmentation. It uses ground-truth masks with matting and inpainting to crop foreground instances and paste them back into the same image. In the first variant, random jittering samples small translations, scales, and rotations near the original position, yielding 1.7 mAP improvement on COCO and 3.3 mAP on Pascal VOC for Mask R-CNN. In the second variant, a location probability map is built from local appearance consistency (weighted RGB contour rings), and paste locations are sampled from this heatmap, giving further gains and reaching 37.9 mAP on COCO test-dev with R101-Mask R-CNN, a total 2.2 mAP over baseline. The method is simple, online, requires no extra training, and the code is released. The core claims are that random jittering helps substantially and that the appearance consistency heatmap provides a further, non-tautological improvement over random placement.
Significance. If the reported gains are robust, this is a practical and generally applicable augmentation that improves several Mask R-CNN variants and the Cascade R-CNN baseline on both COCO and VOC, while adding only about 2% training time. The paper ships released code and models, and it includes useful ablations: a random-paste baseline, hyperparameter sensitivity analysis, and a comparison with a learned context model. The conceptual contribution is a cheap appearance-based location prior that avoids the computational cost of a learned context model. The main weakness is that the unique contribution of the appearance heatmap beyond random jittering rests on small single-run differences and a confounded ablation, so the significance depends on whether those gaps are reproducible and properly isolated.
major comments (3)
- [Sec. 4.4, Tables 1-3] The claimed benefit of the appearance consistency heatmap over plain random jittering rests on small single-run differences: 37.9 vs 37.4 for R101-Mask R-CNN, 36.0 vs 35.5 for R50, and 42.73 vs 42.15 on VOC. No error bars or multiple seeds are reported. Given the stochasticity of training Mask R-CNN, these gaps may be within run-to-run variation, so the central claim that the probability map guidance is responsible for the improvement is not yet supported. Please report the mean and standard deviation over at least three seeds for the vanilla, jitter, and map-guided variants, or otherwise establish statistical significance.
- [Sec. 4.5, Table 6] The random-paste baseline does not isolate the appearance-weighting mechanism. Random paste samples anywhere on the image, whereas random InstaBoost restricts translations to a 1/15-of-object-size neighborhood and map-guided sampling uses the heatmap over the whole resized image. Thus the comparison in Table 6 confounds two changes: the set of allowed translations (support) and the weighting by local appearance. The conclusion that "appearance consistency guidance is essential" is not supported by this experiment. Add a controlled ablation that uses the same candidate location set for both conditions (e.g., all non-overlapping valid positions) and varies only the sampling distribution (uniform vs. heatmap-weighted), while holding the number of pastes and the scale/rotation ranges constant. Without this, the map-guided gain over jitter cannot be attributed to the heatmap rather than to the larger translation search space.
- [Sec. 3.3.1, Eqs. (8)-(9)] The heatmap construction is underspecified, which affects reproducibility of the core novelty. First, the radii at which the three fixed-width contour rings are placed are not defined in the text or in the parameter list. Second, the handling of ring pixels that fall outside the image is described as "we consider the semantic consistency distance of this pixel equals to infinity (and therefore ignored)", which is contradictory: if a pixel is ignored, it should be excluded from the sum, not treated as an infinite distance. Third, because d(D0,D0)=0, Eq. (9) evaluates to +inf at the original position, yet Eq. (4) states the identity transform should have the highest probability; the normalization or truncation used in practice is not stated. Please specify the ring geometry, the out-of-image treatment, and the practical normalization of the heatmap.
minor comments (6)
- [Sec. 3.2] "Parallelly provides additional supervision" should be "simultaneously provides additional supervision" or "in parallel provides".
- [Table 3] The table header repeats "APbb" ("APbb APbb50 APbb75"); the duplicated label should be corrected.
- [Sec. 3.3.3] The statement that the brute-force computation is O(W^2 H^2) applies to scanning all candidate centers for one object; clarify the per-object complexity and how the resizing to a fixed size reduces it.
- [Sec. 4.3] The fixed size (180,120) for heatmap acceleration is a hyperparameter whose sensitivity is not analyzed; a sentence on how this choice affects quality or speed would be useful.
- [Fig. 6] The caption says InstaBoost "guarantees finer instance segmentation result"; such a guarantee cannot be drawn from selected examples, and the wording should be softened to "shows" or "is observed to".
- [Sec. 3.3.2] The term "probability map" is used loosely: after normalization the heatmap is treated as a discrete probability distribution over locations, but the connection to the continuous density fxy(tx,ty) in Eq. (6) should be stated more precisely, especially how normalization is performed.
Circularity Check
No circularity: the heatmap is computed from raw RGB pixels, the method is evaluated on external benchmarks, and the random-paste ablation shows the proposed guidance is not a tautology.
full rationale
The paper's derivation chain is self-contained rather than circular. The probability map in Eq. (3) is posed as a general density over transformation tuples; Eq. (4) is an explicit modeling assumption that the identity transform is most probable; random InstaBoost samples from a neighborhood of that identity transform; and the appearance consistency heatmap is computed directly from RGB contour-ring distances in Eqs. (7)-(8), normalized by Eq. (9), and then sampled via Monte Carlo. None of these quantities is defined in terms of the reported mAP values, and no parameter is fitted to the COCO/VOC test results that are later used as the evaluation. The central claim — that map-guided paste locations outperform random jittering — is empirically checked against external benchmarks, and Table 6 specifically shows that random paste degrades performance relative to the vanilla baseline, demonstrating that the appearance guidance is not equivalent to its input by construction. Self-citations in the paper (e.g., refs. 18, 19, 27, 29, 43, 44) are used only as related work or implementation references and are not load-bearing for the method's derivation. The remaining concerns are experimental: the map-guided gain over jitter is a small single-run delta and the ablation in Table 6 does not fully isolate translation support from appearance weighting. Those are validity or robustness issues, not circularity, so the appropriate score is 0.
Assumptions & free parameters
free parameters (7)
- Translation ratio =
1/15 of object width and height
- Scale range =
0.8 to 1.2
- Rotation range =
[-5, 5] degrees
- Contour widths =
5 pixels per contour
- Contour weights =
0.4, 0.35, 0.25 from inside to outside
- Heatmap resize size =
(180, 120)
- Training schedule =
48 epochs
assumptions (6)
- domain assumption Images are locally continuous and redundant in pixel level, so small jitters around the original object position yield valid training images.
- domain assumption The original object position is the mode of the placement distribution: argmax P(x,y,s,r) = (x0,y0,1,0).
- domain assumption Translation, scale, and rotation are independent: P(x,y,s,r) = fxy(tx,ty) fs(s) fr(r).
- domain assumption Appearance consistency measured on three fixed-width RGB contour rings is a sufficient proxy for feasible placement.
- domain assumption Matting and inpainting produce clean object patches and background holes without harmful artifacts.
- domain assumption Single-run COCO test-dev mAP comparisons are reliable indicators of the true improvement.
Cite this review
Pith. "Pith review of InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting." pith.science (2026). https://pith.science/paper/V57JUVWL
@misc{pith2026190807801,
author = {Pith},
title = {Pith review of: InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/V57JUVWL}},
note = {Machine review of arXiv:1908.07801}
}
read the original abstract
Instance segmentation requires a large number of training samples to achieve satisfactory performance and benefits from proper data augmentation. To enlarge the training set and increase the diversity, previous methods have investigated using data annotation from other domain (e.g. bbox, point) in a weakly supervised mechanism. In this paper, we present a simple, efficient and effective method to augment the training set using the existing instance mask annotations. Exploiting the pixel redundancy of the background, we are able to improve the performance of Mask R-CNN for 1.7 mAP on COCO dataset and 3.3 mAP on Pascal VOC dataset by simply introducing random jittering to objects. Furthermore, we propose a location probability map based approach to explore the feasible locations that objects can be placed based on local appearance similarity. With the guidance of such map, we boost the performance of R101-Mask R-CNN on instance segmentation from 35.7 mAP to 37.9 mAP without modifying the backbone or network structure. Our method is simple to implement and does not increase the computational complexity. It can be integrated into the training pipeline of any instance segmentation model without affecting the training and inference efficiency. Our code and models have been released at https://github.com/GothicAi/InstaBoost
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/ roytseng-tw/Detectron.pytorch, 2018
Detectron.pytorch. https://github.com/ roytseng-tw/Detectron.pytorch, 2018
work page 2018
-
[2]
Mscoco detection leaderboard. http://cocodataset. org/#detection-leaderboard, 2018
work page 2018
- [3]
-
[4]
Whats the point: Semantic segmentation with point supervision
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. Whats the point: Semantic segmentation with point supervision. In ECCV, 2016
work page 2016
-
[5]
Navier-stokes, fluid dynamics, and image and video inpainting
Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. Navier-stokes, fluid dynamics, and image and video inpainting. In CVPR, 2001
work page 2001
-
[6]
Watershed-based segmen- tation and region merging
Andr ´e Bleau and L Joshua Leon. Watershed-based segmen- tation and region merging. Computer Vision and Image Un- derstanding, 2000
work page 2000
-
[7]
Cascade r-cnn: Delving into high quality object detection
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018
2018
-
[8]
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaox- iao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. mmdetection. https://github.com/open-mmlab/ mmdetection, 2018
work page 2018
Show all 45 references
-
[9]
Masklab: Instance segmentation by refining object detection with semantic and direction features
Liang-Chieh Chen, Alexander Hermans, George Papan- dreou, Florian Schroff, Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. In CVPR, 2018
2018
-
[10]
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016
2016
-
[11]
Instance-sensitive fully convolutional networks
Jifeng Dai, Kaiming He, Yi Li, Shaoqing Ren, and Jian Sun. Instance-sensitive fully convolutional networks. In ECCV, 2016
2016
-
[12]
Boxsup: Exploit- ing bounding boxes to supervise convolutional networks for semantic segmentation
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploit- ing bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV, 2015
2015
-
[13]
Instance-aware se- mantic segmentation via multi-task network cascades
Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware se- mantic segmentation via multi-task network cascades. In CVPR, 2016
2016
-
[14]
Learning segmentation masks with the independence prior
Songmin Dai, Xiaoqiang Li, Lu Wang, Pin Wu, Weiqin Tong, and Yimin Chen. Learning segmentation masks with the independence prior. arXiv:1811.04682, 2018
2018 arXiv
-
[15]
Mod- eling visual context is key to augmenting object detection datasets
Nikita Dvornik, Julien Mairal, and Cordelia Schmid. Mod- eling visual context is key to augmenting object detection datasets. In ECCV, 2018
2018
-
[16]
Cut, paste and learn: Surprisingly easy synthesis for instance de- tection
Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance de- tection. In ICCV, 2017
2017
-
[17]
The pascal visual object classes (voc) challenge
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010
2010
-
[18]
Weakly and semi supervised human body part parsing via pose-guided knowledge trans- fer
Hao-Shu Fang, Guansong Lu, Xiaolin Fang, Jianwen Xie, Yu-Wing Tai, and Cewu Lu. Weakly and semi supervised human body part parsing via pose-guided knowledge trans- fer. CVPR, 2018
2018
-
[19]
Rmpe: Regional multi-person pose estimation
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In ICCV, 2017
2017
-
[20]
Contour integration by the human visual system: evidence for a local association field
David J Field, Anthony Hayes, and Robert F Hess. Contour integration by the human visual system: evidence for a local association field. Vision research, 1993
1993
-
[21]
Look into person: Self-supervised structure- sensitive learning and a new benchmark for human parsing
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self-supervised structure- sensitive learning and a new benchmark for human parsing. In CVPR, 2017
2017
-
[22]
Semantic contours from inverse detectors
Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011
2011
-
[23]
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, 2017
2017
-
[24]
A global sampling method for alpha matting
Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. A global sampling method for alpha matting. In CVPR, 2011
2011
-
[25]
Cooperative holistic scene understanding: Unifying 3d object, layout, and cam- era pose estimation
Siyuan Huang, Siyuan Qi, Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout, and cam- era pose estimation. In NIPS, 2018
2018
-
[26]
Lucid data dreaming for video object seg- mentation
Anna Khoreva, Rodrigo Benenson, Eddy Ilg, Thomas Brox, and Bernt Schiele. Lucid data dreaming for video object seg- mentation. International Journal of Computer Vision, pages 1–23
-
[27]
Crowdpose: Efficient crowded scenes pose estimation and a new benchmark
Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. In CVPR, 2019
2019
-
[28]
Fully convolutional instance-aware semantic segmentation
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware semantic segmentation. In CVPR, 2017
2017
-
[29]
Transferable interactiveness knowledge for human-object interaction de- tection
Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction de- tection. In CVPR, 2019
2019
-
[30]
Scribblesup: Scribble-supervised convolutional networks for semantic segmentation
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, 2016
2016
-
[31]
Complex background subtraction by pursuing dynamic spatio-temporal models
Liang Lin, Yuanlu Xu, Xiaodan Liang, and Jianhuang Lai. Complex background subtraction by pursuing dynamic spatio-temporal models. TIP, 2014
2014
-
[32]
Feature pyramid networks for object detection
Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017
2017
-
[33]
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014
2014
-
[34]
See more, know more: Unsuper- vised video object segmentation with co-attention siamese networks
Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsuper- vised video object segmentation with co-attention siamese networks. In CVPR, 2019
2019
-
[35]
Learn- ing to segment object candidates
Pedro O Pinheiro, Ronan Collobert, and Piotr Doll ´ar. Learn- ing to segment object candidates. In NIPS, 2015
2015
-
[36]
Learning human-object interactions by graph parsing neural networks
Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In CVPR, 2018
2018
-
[37]
Faster R-CNN: Towards real-time object detection with re- gion proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with re- gion proposal networks. In NeurIPS, 2015
2015
-
[38]
Segmentation of un- known objects in indoor environments
Andreas Richtsfeld, Thomas M ¨orwald, Johann Prankl, Michael Zillich, and Markus Vincze. Segmentation of un- known objects in indoor environments. In IROS, 2012
2012
-
[39]
Find and focus: Retrieve and localize video events with natural language queries
Dian Shao, Yu Xiong, Yue Zhao, Qingqiu Huang, Yu Qiao, and Dahua Lin. Find and focus: Retrieve and localize video events with natural language queries. In ECCV, 2018
2018
-
[40]
Visual perception: An introduction
Nicholas Wade and Mike Swanston. Visual perception: An introduction. 2013
2013
-
[41]
Salient object detection in the deep learning era: An in-depth survey
Wenguan Wang, Qiuxia Lai, Huazhu Fu, Jianbing Shen, and Haibin Ling. Salient object detection in the deep learning era: An in-depth survey. arXiv preprint arXiv:1904.09146, 2019
1904 arXiv
-
[42]
Saliency-aware video object segmentation
Wenguan Wang, Jianbing Shen, Ruigang Yang, and Fatih Porikli. Saliency-aware video object segmentation. TPAMI, 40(1):20–33, 2017
2017
-
[43]
Srda: Generat- ing instance segmentation annotation via scanning, reason- ing and domain adaptation
Wenqiang Xu, Yonglu Li, and Cewu Lu. Srda: Generat- ing instance segmentation annotation via scanning, reason- ing and domain adaptation. In ECCV, 2018
2018
-
[44]
Ex- plicit shape encoding for real-time instance segmentation
Wenqiang Xu, Haiyang Wang, Fubo Qi, and Cewu Lu. Ex- plicit shape encoding for real-time instance segmentation. arXiv:1908.04067, 2019
1908 arXiv
-
[45]
A stochastic gram- mar of images
Song-Chun Zhu, David Mumford, et al. A stochastic gram- mar of images. Foundations and Trends R© in Computer Graphics and Vision, 2007
2007
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.