REVIEW 4 major objections 6 minor 31 references
FMG-Det: Foundation Model Guided Robust Object Detection
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read FMG-Det uses SAM and CLIP to fix noisy bounding boxes before training and keeps detectors accurate at high noise levels.
desk verdict A genuinely clever, detector-agnostic correction pipeline, but the headline robustness claims rest on a noise protocol whose severity may not be monotonic. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Foundation Model Correction (FMC) pipeline, a training-free preprocessing step. For each noisy box it produces a diverse set of candidate masks by prompting SAM with both the box and its centerpoint; it scores each candidate by combining SAM's score with a CLIP image-text similarity against the class label; it selects the argmax, converts the winning mask to an axis-aligned box, and rejects corrections whose IoU with the original is below $\lambda = 0.05$. This pipeline is what carries the robustness gain, since the ablation attributes most of the improvement to it (VOC MAE falls from 36.4 to 17.5). A secondary mechanism is the learned instance-interpolation module, which uses the detector backbone's ROI features to predict a mixing coefficient $\gamma$ between each noisy and corrected box, letting the training procedure fall back toward the original annotation when the correction is suspect.
What would settle it
Run FMG-Det on a dataset where a large share of noisy boxes have zero overlap with the object, using the stock SAM model; if the detector's mAP at high noise is not better than the same detector trained on the untouched noisy boxes, the claim that SAM/CLIP correction carries the robustness gain is falsified. A cheaper check is to replace SAM's candidate masks with random masks on VOC: the reported gap over SSD-Det should disappear if the masks are the actual cause.
Extended reading notes
Core claim
The paper claims that noisy bounding boxes can be corrected well enough by foundation models, without any training, to serve as high-quality training signal for a detector. The correction procedure (FMC) generates candidate masks by calling SAM twice per annotation—once with the noisy box as a prompt and once with its centerpoint—then computes a CLIP score for each masked crop against the class label, combines it with SAM's own mask score via $0.5 c_{j,k} + 0.5 s_{j,k}$, selects the maximum, converts the mask to a box, and discards the correction if its IoU with the noisy box falls below $\lambda = 0.05$. FMG-Det then feeds both the noisy and corrected boxes to OA-MIL and trains a small network to output $\gamma = \sigma(G_\theta([\mathrm{ROI}(\hat{b}_j,x); \mathrm{ROI}(b_j,x)]))$, producing the interpolated box $b^*_j = \gamma \hat{b}_j + (1-\gamma) b_j$. Under the synthetic noise protocol described in Section 3, the method attains 46.5 mAP on VOC 2007 at noise level 1.0 versus 31.3 for SSD-Det and 19.0 for Faster R-CNN, and 15.7 mAP on COCO at noise 0.8 versus 1.5 for SSD-Det, cutting the paper's mean-absolute-error metric by 6.6 points on VOC and 6.8 on COCO. It also reports the first few-shot robust-detection results, where noise is shown to be more damaging.
Load-bearing premise
The entire robustness gain rests on the assumption that SAM, when prompted with a noisy box and its centerpoint, produces a candidate mask that actually covers the target object; when that fails, the pipeline keeps the original noisy annotation and the main source of improvement is gone.
Editorial extensions
If this is right
- At noise level 1.0 on PASCAL VOC, the corrected pipeline reports 46.5 mAP while the prior best baseline reports 31.3, so high-noise annotation sets that were previously unusable become trainable.
- On COCO at noise level 0.8, FMG-Det reports 15.7 mAP against 1.5 for SSD-Det, meaning the method keeps detection viable where previous approaches collapsed below 2 mAP.
- In few-shot settings, noise is more damaging and FMG-Det still beats the prior best, suggesting the correction pipeline helps most when every training example counts.
- Because the correction stage is offline and detector-agnostic, the method can be paired with any two-stage detector and adds minimal training-time computation.
- The ablation indicates the FMC pipeline alone already reaches state-of-the-art VOC robustness, with the interpolation module adding a further improvement.
Reading between the lines
- The paper's own limitation note implies that when a noisy box has no overlap with the object, SAM tends to segment background or a neighbor, and the $\lambda$ filter then reverts to the noisy annotation; one can infer that on datasets where such zero-overlap errors are common, the expected gain would shrink toward zero or reverse.
- Because the synthetic noise is uniform and random, real annotator noise—which tends to be systematic—may be easier for the same pipeline to correct; that would make the reported gains a lower bound, but the paper does not test this.
- One could test the interpolation module directly: on corrected boxes that overlap the object, learned $\gamma$ should be near 1, and on rejected corrections, near 0; the paper does not report these values.
- The method's reliance on SAM suggests a domain-shift test: applying the unmodified pipeline to medical or aerial imagery should fail unless SAM is swapped for a domain-tuned variant, matching the paper's stated limitation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FMG-Det, a method for training object detectors when bounding-box annotations are noisy. It combines an offline Foundation Model Correction (FMC) pipeline, which uses SAM and CLIP to generate and select corrected boxes from noisy ones, with the OA-MIL multiple-instance-learning framework and a learned instance-interpolation module. The method is evaluated on PASCAL VOC and MS COCO under a synthetic bounding-box noise protocol, in both fully supervised and few-shot settings. The authors report large MAE reductions compared with Faster R-CNN, OA-MIL, and SSD-Det, and they claim average gains of 6.6 mAP on VOC and 6.8 mAP on COCO.
Significance. If the reported results hold, FMG-Det offers a simple, detector-agnostic pre-processing step that could substantially improve robustness to annotation noise, especially in few-shot settings, and the paper provides an honest ablation study and an explicit limitations section. The central weakness is that the strongest evidence is concentrated on the VOC full-training sweep, while the COCO evidence is thin and the synthetic severity scale appears non-monotonic in the reported numbers. Because the headline claims are framed through the MAE metric, the non-monotonicity directly affects whether the claimed gains are a controlled stress test.
major comments (4)
- [Section 3 and Table 1] The reported results are inconsistent with a monotone noise severity scale, which the MAE metric requires. For Faster R-CNN, mAP at noise 0.8 is 13.5 +/- 0.12 but at noise 1.0 it is 19.0 +/- 0.52; OA-MIL similarly rises from 16.2 +/- 0.25 to 18.2 +/- 0.40. Since the paper defines MAE as the average absolute drop relative to the noise-free baseline across levels, these entries imply that level 1.0 is not strictly more damaging than level 0.8. The authors should report per-level statistics such as the IoU distribution between noisy and clean boxes and the fraction of boxes clipped or discarded after perturbation, or otherwise justify the severity ordering; without this, the 'severe noise' comparisons and the MAE gains are not on solid ground.
- [Table 2 (MS COCO)] The COCO experiment reports MAE values but omits the noise-0.0 baseline mAP for every model and evaluates at only two noise levels. Because the abstract's 6.8-point COCO gain is based on this MAE, the central COCO claim cannot be checked from the paper. The authors should add the 0.0 column, error bars over runs, and ideally intermediate noise levels; the current two-point evaluation is too thin to support the headline comparison.
- [Section 3] The noise injection protocol is under-specified. The text says 'Values Δx, Δy, Δw, and Δh are sampled from a uniform distribution on the interval (−0.0, 1.0)' but does not state how the noise level parameter (0.0–1.0) maps to the sampling interval, nor whether coordinates are clipped or boxes discarded when they fall outside the image. This makes the experiments non-reproducible and is likely connected to the non-monotonicity reported in Table 1. A precise formula or pseudocode for the noise generation is needed.
- [Abstract and Section 5.1.1] The abstract states an 'average performance gain of 6.8 mAP on COCO and 6.6 mAP on VOC', but Tables 1–2 report reductions in MAE (the average drop in mAP), not average mAP increases. These are different quantities, and the wording should be corrected to avoid claiming a gain in mean average precision that the experiments do not directly report.
minor comments (6)
- [General] There are several typographical and formatting issues, including 'FOUNDA TION' in the title, 'VA' in the affiliation line, and 'stocastic' in the Limitations section; these should be cleaned up.
- [Table 2] Table 2 reports no variance or error bars, unlike Table 1 which does; this makes it difficult to assess the reliability of the COCO results, especially the 15.7 mAP entry at noise 0.8.
- [Section 4.2.1] The instance interpolation module uses the detector's ROI extractor and backbone features, but it is not clear whether gradients flow through the ROI features and backbone during training of Gθ or only through the small MLP; this should be clarified.
- [Supplementary Table 3] The few-shot results exhibit the same non-monotonic pattern as Table 1, e.g., 1-shot Faster R-CNN gets 0.4 mAP at noise 0.8 and 1.7 mAP at noise 1.0; a dedicated analysis of why the noise scale is not monotonic would strengthen the paper.
- [Section 4.1.3] The λ = 0.05 IoU threshold for accepting a correction is very permissive and was selected 'empirically running on subsets of the data'; a sensitivity analysis over α and λ, or at least a statement of how the held-out test set was protected from this selection, would improve confidence in the method's hyperparameter choices.
- [Reproducibility] The paper references the OA-MIL and MMDetection repositories but does not provide code or a complete list of few-shot seeds; releasing the code and seed details would make the few-shot results reproducible.
Circularity Check
No significant circularity: FMG-Det's robustness gains are measured externally against frozen SAM/CLIP and prior baselines, with no load-bearing self-citation or fitted prediction.
full rationale
FMG-Det makes an empirical claim: using frozen SAM and CLIP to correct noisy boxes, followed by a learned interpolation between corrected and noisy boxes, retains mAP on VOC and COCO under a synthetic noise protocol. I checked the derivation chain for the seven circularity patterns. First, no self-definition is present: Eq. (1) selects a mask by a weighted sum of SAM and CLIP scores, and the selected box is not defined in terms of the reported mAP; Eqs. (2)-(3) define the interpolation as a learned convex combination, with performance measured externally on fixed benchmarks. Second, no fitted input is called a prediction: the only tuned scalars, alpha = 0.5 and lambda = 0.05, are disclosed in Section E.1 as hyperparameters chosen empirically on data subsets to minimize dramatically shifted boxes; they do not algebraically determine the COCO/VOC MAE numbers, and the paper does not present them as predictions. Third, no load-bearing self-citations exist: the references supporting the method, OA-MIL [2], SSD-Det [3], and the SAM and CLIP foundation models, are all external prior work, not the authors' own results. Fourth, no uniqueness theorem or ansatz is imported from the authors' own prior publications. Fifth, the contribution is not a renaming of a known result; it is a concrete, tested pipeline combining SAM, CLIP, and MIL. The Limitations section F explicitly states that when SAM fails the method reverts to the noisy input, and that no naturally noisy public dataset was tested; this is an honest scope limitation, not circularity. The non-monotonic VOC baseline at noise 0.8 versus 1.0 (Faster R-CNN: 13.5 versus 19.0 mAP) is a legitimate internal correctness concern about the synthetic noise protocol, but it does not make FMG-Det's result reduce to its own inputs. Therefore, no circular step is present.
Assumptions & free parameters
free parameters (3)
- alpha (SAM/CLIP score mixing weight) =
0.5
- lambda (IoU acceptance threshold for corrections) =
0.05
- gamma-network architecture (3-layer ReLU MLP) =
3 feed-forward layers
assumptions (5)
- domain assumption Synthetic independent per-coordinate uniform noise models real annotation noise (Section 3, following [2]).
- domain assumption Class labels are accurate; only box coordinates are noisy (Section 3).
- domain assumption SAM, prompted by noisy boxes and centerpoints, yields masks that cover the true object (Section 4.1.1).
- domain assumption CLIP image-text similarity scores rank mask quality for box tightness (Section 4.1.2).
- domain assumption The true box lies near the convex combination of noisy and corrected boxes (Section 4.2.1, Eq. 2).
Cite this review
Pith. "Pith review of FMG-Det: Foundation Model Guided Robust Object Detection." pith.science (2026). https://pith.science/paper/S75KHQ4V
@misc{pith2026250523726,
author = {Pith},
title = {Pith review of: FMG-Det: Foundation Model Guided Robust Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/S75KHQ4V}},
note = {Machine review of arXiv:2505.23726}
}
read the original abstract
Collecting high quality data for object detection tasks is challenging due to the inherent subjectivity in labeling the boundaries of an object. This makes it difficult to not only collect consistent annotations across a dataset but also to validate them, as no two annotators are likely to label the same object using the exact same coordinates. These challenges are further compounded when object boundaries are partially visible or blurred, which can be the case in many domains. Training on noisy annotations significantly degrades detector performance, rendering them unusable, particularly in few-shot settings, where just a few corrupted annotations can impact model performance. In this work, we propose FMG-Det, a simple, efficient methodology for training models with noisy annotations. More specifically, we propose combining a multiple instance learning (MIL) framework with a pre-processing pipeline that leverages powerful foundation models to correct labels prior to training. This pre-processing pipeline, along with slight modifications to the detector head, results in state-of-the-art performance across a number of datasets, for both standard and few-shot scenarios, while being much simpler and more efficient than other approaches.
Reference graph
Works this paper leans on
-
[1]
FMG-Det: Foundation Model Guided Robust Object Detection
INTRODUCTION Collecting high quality annotations is challenging for object detection tasks; annotators must precisely and consistently place bounding boxes around the target objects. When ob- jects sharply contrast with the background and have a rela- tively uniform shape, there is unlikely to be much variation in the specified locations. However, when ob...
work page Pith review arXiv 2025
-
[2]
RELA TED WORK Initial work in robust object detection focused on scenarios where both the bounding boxes and the class labels contained noise [8, 9, 10, 11]. Zhang et al. [12] kicked off the line of work most similar to our own, exclusively focusing on bound- ing box noise. Liu et al. [2] took this work further, explicitly demonstrating that bounding box ...
-
[3]
TASK DESCRIPTION Consider a standard object detection task where a model is given an image i and is tasked with producing a set of bound- ing boxes Bi and corresponding set of labels Ci for each tar- get object in the image. Each bounding box bi consists of four coordinates (x, y, w, h) representing the center coordi- nates and width/height of a box that ...
-
[4]
METHOD 4.1. Foundation Model Correction Pipeline We first propose a pre-processing methodology that directly adjusts the target bounding box to the target object, improv- ing the quality of the training data by rectifying any poten- tial errors that have been made and providing more consistent annotations to train on. We call this our Foundation Model Cor...
work page 2022
-
[5]
EXPERIMENTS Following prior work [2, 4, 3], we evaluate our model on MS- COCO [1] and PASCAL VOC [5] and synthetically add noise using the procedure outlined in Section 3. Since these splits are fully aligned with prior works, we do not exhaustively run all baseline models as all of the results that we present are directly comparable. We do rerun current ...
-
[6]
CONCLUSION Object detector performance degrades significantly if bound- ing boxes are not carefully placed over the target objects. We presented a new robust object detection algorithm, FMG-Det, that leverages powerful foundation models to correct bound- ing boxes, mitigating the impact of noisy annotations. We demonstrated through rigorous experimentatio...
-
[7]
Learning trans- ferable visual models from natural language supervi- sion,
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning trans- ferable visual models from natural language supervi- sion,” in ICML, 2021
work page 2021
-
[8]
Microsoft coco: Common objects in context,
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick, “Microsoft coco: Common objects in context,” in ECCV. Springer, 2014, pp. 740–755
work page 2014
Show all 31 references
-
[9]
Robust object detection with inaccurate bounding boxes,
Chengxin Liu, Kewei Wang, Hao Lu, Zhiguo Cao, and Ziming Zhang, “Robust object detection with inaccurate bounding boxes,” in ECCV. Springer, 2022, pp. 53–69
2022
-
[10]
Spatial self-distillation for object detection with inaccurate bounding boxes,
Di Wu, Pengfei Chen, Xuehui Yu, Guorong Li, Zhen- jun Han, and Jianbin Jiao, “Spatial self-distillation for object detection with inaccurate bounding boxes,” in ICCV, 2023, pp. 6855–6865
2023
-
[11]
Distribution- aware calibration for object detection with noisy bound- ing boxes,
Donghao Zhou, Jialin Li, Jinpeng Li, Jiancheng Huang, Qiang Nie, Yong Liu, Bin-Bin Gao, Qiong Wang, Pheng-Ann Heng, and Guangyong Chen, “Distribution- aware calibration for object detection with noisy bound- ing boxes,” arXiv preprint, 2023
2023
-
[12]
The pascal visual object classes (voc) challenge,
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman, “The pascal visual object classes (voc) challenge,” IJCV, vol. 88, pp. 303–338, 2010
2010
-
[13]
Segment anything,
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick, “Segment anything,” arXiv preprint, 2023
2023
-
[14]
[15] are also closely related works that explicitly focus on remote sensing
and Zhu et al. [15] are also closely related works that explicitly focus on remote sensing. Object-centric representation learning is a related field, where a model is trained to localize objects in a fully unsu- pervised setting [16, 17, 18]. These models must also learn to h...
-
[15]
Training object detectors with noisy data,
Simon Chadwick and Paul Newman, “Training object detectors with noisy data,” in 2019 IEEE Intelligent Ve- hicles Symposium (IV). IEEE, 2019, pp. 1319–1325
2019
-
[16]
Towards robust adaptive object detec- tion under noisy annotations,
Xinyu Liu, Wuyang Li, Qiushi Yang, Baopu Li, and Yixuan Yuan, “Towards robust adaptive object detec- tion under noisy annotations,” in CVPR, 2022
2022
-
[17]
Towards noise-resistant object detection with noisy annotations,
Junnan Li, Caiming Xiong, Richard Socher, and Steven Hoi, “Towards noise-resistant object detection with noisy annotations,” arXiv preprint, 2020
2020
-
[18]
Noisy annotation refinement for object detec- tion,
Jiafeng Mao, Qing Yu, Yoko Yamakata, and Kiyoharu Aizawa, “Noisy annotation refinement for object detec- tion,” BMVC, 2021
2021
-
[19]
Learn- ing to localize objects with noisy labeled instances,
Xiaopeng Zhang, Yang Yang, and Jiashi Feng, “Learn- ing to localize objects with noisy labeled instances,” in AAAI, 2019, vol. 33, pp. 9219–9226
2019
-
[20]
A novel benchmark for refinement of noisy localization labels in autolabeled datasets for object detection,
Andreas B ¨ar, Jonas Uhrig, Jeethesh Pai Umesh, Mar- ius Cordts, and Tim Fingscheidt, “A novel benchmark for refinement of noisy localization labels in autolabeled datasets for object detection,” inCVPR, 2023, pp. 3851– 3860
2023
-
[21]
Dynamic loss decay based robust oriented object detec- tion on remote sensing images with noisy labels,
Guozhang Liu, Ting Liu, Mengke Yuan, Tao Pang, Guangxing Yang, Hao Fu, Tao Wang, and Tongkui Liao, “Dynamic loss decay based robust oriented object detec- tion on remote sensing images with noisy labels,” arXiv preprint, 2024
2024
-
[22]
Robust tiny object detection in aerial images amidst label noise,
Haoran Zhu, Chang Xu, Wen Yang, Ruixiang Zhang, Yan Zhang, and Gui-Song Xia, “Robust tiny object detection in aerial images amidst label noise,” arXiv preprint, 2024
2024
-
[23]
Discover- ing objects that can move,
Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert, “Discover- ing objects that can move,” in CVPR, 2022, pp. 11789– 11798
2022
-
[24]
Bridging the gap to real-world object-centric learning,
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Sch ¨olkopf, Thomas Brox, et al., “Bridging the gap to real-world object-centric learning,” in ICLR, 2023
2023
-
[25]
Sim- ple unsupervised object-centric learning for complex and naturalistic videos,
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn, “Sim- ple unsupervised object-centric learning for complex and naturalistic videos,” NeurIPS, vol. 35, pp. 18181– 18196, 2022
2022
-
[26]
Grounded sam: Assembling open-world models for di- verse visual tasks,
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang, “Grounded sam: Assembling open-world models for di- verse visual tasks,” 2024
2024
-
[27]
Frustratingly simple few-shot object detection,
Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu, “Frustratingly simple few-shot object detection,” in ICML, 2020, pp. 9919–9928
2020
-
[28]
Faster r-cnn: Towards real-time object detection with region proposal networks,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,”NeurIPS, vol. 28, 2015
2015
-
[29]
MMDetection: Open mmlab detection toolbox and benchmark,
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang,...
2019
-
[30]
Deep residual learning for image recognition,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778
2016
-
[31]
Segment anything in medical images,
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang, “Segment anything in medical images,” Nature Communications, vol. 15, pp. 1–9, 2024. 0.0 0.2 0.4 0.6 0.8 1.0 Bounding Box Noise 0 10 20 30 40 50 60 70 80mAP Faster RCNN OA-MIL SSD-Det FMG-Det (Ours) Fig. 4. VOC te...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.