REVIEW 3 major objections 4 minor 17 references
Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Fine-tuned YOLO11 Nano outperforms open-vocabulary models for MEP detection on construction sites.
desk verdict New robot-collected MEP dataset and benchmark showing fine-tuned YOLO beats zero-shot open-vocabulary models by a huge margin, but the open-vocabulary baselines are under-specified (no prompts, thresholds, or dataset URL). read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The benchmark itself is the mechanism: a single robot-collected dataset, split into 70% training, 20% validation, and 10% testing, with 10 MEP classes and 14,064 annotated instances, and a fixed evaluation protocol of precision, recall, and F1 at 0.5 intersection over union on the test split. The central comparison is YOLO11 Nano, a closed-set detector fine-tuned for 200 epochs, versus three open-vocabulary models — Grounded SAM2, Grounding DINO, and DETIC — evaluated in their pre-trained, zero-shot form. Open-vocabulary here means the models can in principle detect objects named in text even if those classes were not seen during training, which is what makes the failure on specialized MEP parts informative. The paper also uses parameter counts to frame the gap: YOLO11 Nano has 2.6 million parameters, while Grounded SAM2 has 910 million, so the accuracy advantage also comes with a large computational-efficiency advantage.
What would settle it
Run the same test images through Grounding DINO, Grounded SAM2, and DETIC again, this time with construction-specific prompts naming each of the 10 MEP classes and with confidence thresholds tuned on the validation split; if any of these models then reaches F1 near YOLO11 Nano's 0.89, the paper's central claim is contradicted. The simplest version: report the prompt texts and thresholds used, and show whether the reported F1 numbers change.
Extended reading notes
Core claim
On a real-world dataset gathered from a teleoperated ground robot moving through an active construction site, the paper shows that a task-specific detector trained with just 2.6 million parameters can detect all ten MEP classes with F1 0.89, precision 0.87, and recall 0.901 at 0.5 IoU. Under the same test conditions, Grounded SAM2, Grounding DINO, and DETIC — all used pre-trained and without fine-tuning — reach F1 0.018, 0.032, and 0.014, and fail entirely on several classes such as cable tray fittings. The paper claims that this gap reflects a current limitation of open-vocabulary models on domain-specific, specialized visual tasks, and that the practical choice for real-time robotic MEP monitoring remains a fine-tuned lightweight detector.
Load-bearing premise
The comparison assumes that running each open-vocabulary model with its default settings and no special wording is a fair test of what it can do for MEP detection.
Editorial extensions
If this is right
- For construction-site MEP monitoring with mobile robots, the results imply that a fine-tuned lightweight detector is the currently reliable option, at least until open-vocabulary models are adapted to the domain.
- Because YOLO11 Nano runs at 23.36 FPS on an NVIDIA Jetson Orin Nano, the approach fits the compute budget of an onboard robot system, whereas the largest open-vocabulary model (910M parameters) is far harder to deploy on such hardware.
- The open-vocabulary models' near-zero scores on classes like cable tray fittings indicate that generic pretraining data (COCO, LVIS) does not transfer to MEP components, so gains will likely require domain-specific data rather than larger generic models alone.
- On the dataset released with the paper, future open-vocabulary models can be measured against these numbers, giving the community a concrete baseline to beat.
Reading between the lines
- The reported gap is specifically against zero-shot, default-configuration use of the open-vocabulary models; the paper does not report prompt texts or confidence thresholds, so a prompt-engineered or lightly fine-tuned version of these models could plausibly narrow the gap on the same test set.
- If that happened, the practical takeaway would shift toward a hybrid: a fast fine-tuned detector for known MEP classes plus an open-vocabulary model as a fallback for novel or rare components.
- Because the dataset is shared, one can track future vision-language models against the same 10 classes; a reasonable target would be matching YOLO11's F1 without per-class fine-tuning, which would amount to a definition of 'ready for MEP detection.'
- The single-site data collection leaves open whether the fine-tuned model's advantage persists across different construction sites, lighting conditions, and camera viewpoints; that cross-site generalization question is not addressed by the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates whether open-vocabulary detection models are ready for detecting MEP (mechanical, electrical, plumbing) components on construction sites. The authors collected a real-world dataset using a mobile ground robot, annotated 10 MEP component classes, and compared a fine-tuned YOLO11 Nano detector against three pre-trained open-vocabulary models (Grounding SAM2, Grounding DINO, and DETIC) used without fine-tuning. The reported results show a very large performance gap: YOLO11 Nano achieves an overall F1 of 0.89, while the open-vocabulary models achieve F1 scores between 0.014 and 0.032 on the same test split. The paper concludes that fine-tuned lightweight models still largely outperform open-vocabulary models in specialized environments and domain-specific tasks.
Significance. If the comparison is taken at face value, the paper provides a useful data point for practitioners in construction robotics: for this specific MEP detection task, a 2.6M-parameter fine-tuned detector massively outperforms much larger zero-shot open-vocabulary models. The main strength is the independently collected, annotated, and publicly shared dataset, which is a valuable resource for future work. The paper also reports deployment-relevant metrics (model size, inference speed on an embedded platform). However, the central quantitative claim rests on an under-specified evaluation protocol: the open-vocabulary models are used zero-shot with undisclosed prompts and confidence thresholds, no error bars are reported, and no attempt is made to tune or adapt the open-vocabulary models. The qualitative direction of the result is likely robust given the size of the gap, but the exact F1 values and the broad 'not ready' framing are conditional on missing details.
major comments (3)
- [Section 2, Evaluation Procedure; Table 2] The evaluation procedure does not report the text prompts, class vocabularies, or detection confidence thresholds used for Grounding SAM2, Grounding DINO, and DETIC. For open-vocabulary detectors these choices strongly control the precision-recall trade-off and can change reported F1 by orders of magnitude. Without this information, the near-zero F1 values in Table 2 cannot be interpreted as a fair representation of what these models can achieve for MEP detection. The manuscript should specify the exact prompt strings (e.g., 'pipe fitting', 'cable tray fitting', or a combined prompt), the box/text thresholds, and any post-processing filters. If the authors used generic prompts or default thresholds, the conclusion should be narrowed to 'default, untuned open-vocabulary models' rather than 'open-vocabulary models are not ready'.
- [Section 3, Results and Discussion; Table 2] No error bars, repeated runs, or statistical significance tests are reported. The test split contains relatively few instances for some classes (e.g., Generator has 124 instances, Cable Tray Fitting has 317), so per-class F1 values such as 0.0 vs 0.005 may not be meaningful. Given the small class sizes and the single train/validation/test split, the authors should provide confidence intervals or at least report the number of true positives/false positives underlying the aggregate F1 scores.
- [Section 4, Conclusion] The conclusion states that fine-tuned models 'significantly outperformed open-vocabulary models in detection accuracy, computational efficiency, and real-time applicability' and that open-vocabulary performance 'remains limited.' This overgeneralizes because the open-vocabulary models were only evaluated zero-shot with undisclosed prompts and thresholds, and no prompt engineering, prompt tuning, or fine-tuning was attempted. The paper's own future-work suggestions (fine-tuning and construction-related prompts) indicate that the evidence supports a narrower claim: a fine-tuned lightweight detector outperforms untuned open-vocabulary models in this specific setup. The abstract and conclusion should be revised to reflect this scope.
minor comments (4)
- [Section 2, Dataset] The statement 'The dataset is shared online' does not include a link or repository identifier; the reader cannot access the data without further searching.
- [Section 3, Real-world Applicability] The manuscript reports inference speed only for YOLO11 Nano (23.36 FPS on Jetson Orin Nano) and does not report inference times for the three open-vocabulary models on the same hardware. The claim that open-vocabulary models are 'less practical' for real-time use would be stronger with at least approximate runtime comparisons.
- [Section 3, Qualitative examples] The sentence 'the model outperformed human annotations' is not supported by a quantitative measure; it may be more precise to say that the model detected objects that the annotators had missed, which is a qualitative observation.
- [Table 1] The row labeled 'All' reports total images as 8885, but the sum of per-class image counts exceeds this number because images contain multiple classes; this is fine, but the table caption should clarify that 'Images' counts are not disjoint across classes.
Circularity Check
No circularity: the paper is an empirical benchmark on an independently collected dataset, with no fitted parameter renamed as a prediction and no load-bearing self-citation.
full rationale
The paper's central claim is that a fine-tuned, lightweight YOLO11 Nano outperforms three pre-trained open-vocabulary detectors on a newly collected and manually annotated MEP dataset. This is an externally grounded empirical comparison, not a derivation from assumptions that encode the conclusion. The fine-tuned model is trained on one split of the dataset and evaluated on a held-out test split; the open-vocabulary models are evaluated zero-shot on the same test split. No equation in the paper reduces the outcome to its inputs, and no fitted parameter is renamed as a prediction. The authors' self-citations, such as reference [2] on robotic deployment considerations, are contextual and not load-bearing for the quantitative comparison. Concerns about undisclosed prompts or confidence thresholds for the open-vocabulary models are evaluation-validity issues, not circularity: they question whether the baseline was configured optimally, but they do not show that any result is true by construction. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (2)
- Detection confidence thresholds for open-vocabulary models =
not reported (likely defaults)
- Prompt templates for open-vocabulary models =
not reported
assumptions (3)
- domain assumption The manually annotated dataset is representative of MEP components on active construction sites and the labels are correct.
- domain assumption The 10% test split from a single construction site supports general conclusions about open-vocabulary model readiness for MEP detection.
- domain assumption Zero-shot evaluation with no prompt tuning or threshold calibration fairly represents the capability of open-vocabulary models for this task.
Cite this review
Pith. "Pith review of Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites." pith.science (2026). https://pith.science/paper/64RGUMPA
@misc{pith2026250109267,
author = {Pith},
title = {Pith review of: Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites},
year = {2026},
howpublished = {\url{https://pith.science/paper/64RGUMPA}},
note = {Machine review of arXiv:2501.09267}
}
read the original abstract
The construction industry has long explored robotics and computer vision, yet their deployment on construction sites remains very limited. These technologies have the potential to revolutionize traditional workflows by enhancing accuracy, efficiency, and safety in construction management. Ground robots equipped with advanced vision systems could automate tasks such as monitoring mechanical, electrical, and plumbing (MEP) systems. The present research evaluates the applicability of open-vocabulary vision-language models compared to fine-tuned, lightweight, closed-set object detectors for detecting MEP components using a mobile ground robotic platform. A dataset collected with cameras mounted on a ground robot was manually annotated and analyzed to compare model performance. The results demonstrate that, despite the versatility of vision-language models, fine-tuned lightweight models still largely outperform them in specialized environments and for domain-specific tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Robotics in the construction industry: State of the art and future opportunities
Guglielmo Carra, Alfredo Argiolas, Alessandro Bel- lissima, Marta Niccolini, and Matteo Ragaglia. Robotics in the construction industry: State of the art and future opportunities. In ISARC. Proceed- ings of the International Symposium on Automation and Robotics in Construction, volume 35, pages 1–8. IAARC Publications, 2018
work page 2018
-
[2]
Robotic deployment on construction sites: considerations for safety and productivity impact
Rafael Gomes Braga, Muhammad Owais Tahir, Ivanka Iordanova, and David St-Onge. Robotic deployment on construction sites: considerations for safety and productivity impact. arXiv preprint arXiv:2404.13143, 2024
work page Pith review arXiv 2024
-
[3]
Chun Wang, Peng Ouyang, Xiaodong Liu, Zhihua Zou, Yuanping Wang, and Ruiling Wang. Future prospects of the assembly model for mep systems in chinese buildings: A whole life cycle perspective. Applied Sciences, 14(15):6818, 2024
work page 2024
-
[4]
An object-based 3d walk-through model for interior construction progress monitoring
Seungjun Roh, Zeeshan Aziz, and Feniosky Pena- Mora. An object-based 3d walk-through model for interior construction progress monitoring. Automa- tion in Construction, 20(1):66–75, 2011
work page 2011
-
[5]
Small and overlapping worker detection at construc- tion sites
Minsoo Park, Jinyeong Bak, Seunghee Park, et al. Small and overlapping worker detection at construc- tion sites. Automation in Construction, 151:104856, 2023
work page 2023
-
[6]
Casnli-yolo: construction site multi-target detection method based on improved yolov5s
Fei Hao, Tianqi Zhang, Guanghong He, Ruofei Dou, and Chao Meng. Casnli-yolo: construction site multi-target detection method based on improved yolov5s. Measurement Science and Technology , 35 (8):085202, 2024
work page 2024
-
[7]
Object detection using yolo: Challenges, architectural successors, datasets and applications
Tausif Diwan, G Anirudh, and Jitendra V Temb- hurne. Object detection using yolo: Challenges, architectural successors, datasets and applications. multimedia Tools and Applications , 82(6):9243– 9275, 2023
work page 2023
-
[8]
Yolo-world: Real- time open-vocabulary object detection
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 16901–16911, 2024
work page 2024
Show all 17 references
-
[9]
Autorepo: A general framework for multimodal llm-based automated construction reporting
Hongxu Pu, Xincong Yang, Jing Li, and Runhao Guo. Autorepo: A general framework for multimodal llm-based automated construction reporting. Expert Systems with Applications, 255:124601, 2024
2024
-
[10]
Omni-scan2bim: A ready-to-use scan2bim approach based on vision foundation models for mep scenes
Boyu Wang, Zhengyi Chen, Mingkai Li, Qian Wang, Chao Yin, and Jack CP Cheng. Omni-scan2bim: A ready-to-use scan2bim approach based on vision foundation models for mep scenes. Automation in Construction, 162:105384, 2024
2024
-
[11]
Constscene: Dataset and model for advancing robust semantic segmentation in construction environments
Maghsood Salimi, Mohammad Loni, Sara Afshar, Antonio Cicchetti, and Marjan Sirjani. Constscene: Dataset and model for advancing robust semantic segmentation in construction environments. arXiv preprint arXiv:2312.16516, 2023
2023 arXiv
-
[12]
Yolov11: An overview of the key architectural enhancements
R Khanam and M Hussain. Yolov11: An overview of the key architectural enhancements. arxiv 2024. arXiv preprint arXiv:2410.17725
2024 arXiv
-
[13]
Grounded sam: As- sembling open-world models for diverse visual tasks
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: As- sembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024
2024 arXiv
-
[14]
Grounding dino: Marrying dino with grounded pre-training for open-set object detection
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jian- wei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vi- sion, pages 38–55. Spri...
2025
-
[15]
Detecting twenty- thousand classes using image-level supervision
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨ uhl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference on Computer Vision , pages 350–368. Springer, 2022
2022
-
[16]
Microsoft coco: Com- mon objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proce...
2014
-
[17]
LVIS: A dataset for large vocabulary instance segmentation
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019
2019
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.