Pith. sign in

REVIEW 3 major objections 4 minor 17 references

Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Fine-tuned YOLO11 Nano outperforms open-vocabulary models for MEP detection on construction sites.

desk verdict New robot-collected MEP dataset and benchmark showing fine-tuned YOLO beats zero-shot open-vocabulary models by a huge margin, but the open-vocabulary baselines are under-specified (no prompts, thresholds, or dataset URL). read the letter →

arxiv 2501.09267 v2 pith:64RGUMPA submitted 2025-01-16 cs.CV cs.RO

classification cs.CVcs.RO
keywords MEPdetectionopen-vocabularyobjectvision-languagemodelsconstructionroboticsYOLO11fine-tuninggroundrobotsitemonitoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Construction-site robots need to spot mechanical, electrical, and plumbing (MEP) components reliably, and open-vocabulary vision-language models promise to do this without task-specific training. This paper tests that promise with a dataset of 8,885 images collected by a ground robot on an active construction site, annotated across 10 MEP classes. A fine-tuned YOLO11 Nano detector with 2.6 million parameters reaches an overall F1 of 0.89, while three pre-trained open-vocabulary detectors — Grounded SAM2, Grounding DINO, and DETIC — used without fine-tuning score 0.018, 0.032, and 0.014. The authors conclude that fine-tuned lightweight detectors still substantially outperform open-vocabulary models in specialized environments, and that the open-vocabulary approach is not yet ready for MEP monitoring on construction sites.

What carries the argument

The benchmark itself is the mechanism: a single robot-collected dataset, split into 70% training, 20% validation, and 10% testing, with 10 MEP classes and 14,064 annotated instances, and a fixed evaluation protocol of precision, recall, and F1 at 0.5 intersection over union on the test split. The central comparison is YOLO11 Nano, a closed-set detector fine-tuned for 200 epochs, versus three open-vocabulary models — Grounded SAM2, Grounding DINO, and DETIC — evaluated in their pre-trained, zero-shot form. Open-vocabulary here means the models can in principle detect objects named in text even if those classes were not seen during training, which is what makes the failure on specialized MEP parts informative. The paper also uses parameter counts to frame the gap: YOLO11 Nano has 2.6 million parameters, while Grounded SAM2 has 910 million, so the accuracy advantage also comes with a large computational-efficiency advantage.

What would settle it

Run the same test images through Grounding DINO, Grounded SAM2, and DETIC again, this time with construction-specific prompts naming each of the 10 MEP classes and with confidence thresholds tuned on the validation split; if any of these models then reaches F1 near YOLO11 Nano's 0.89, the paper's central claim is contradicted. The simplest version: report the prompt texts and thresholds used, and show whether the reported F1 numbers change.

Watch

Extended reading notes

Core claim

On a real-world dataset gathered from a teleoperated ground robot moving through an active construction site, the paper shows that a task-specific detector trained with just 2.6 million parameters can detect all ten MEP classes with F1 0.89, precision 0.87, and recall 0.901 at 0.5 IoU. Under the same test conditions, Grounded SAM2, Grounding DINO, and DETIC — all used pre-trained and without fine-tuning — reach F1 0.018, 0.032, and 0.014, and fail entirely on several classes such as cable tray fittings. The paper claims that this gap reflects a current limitation of open-vocabulary models on domain-specific, specialized visual tasks, and that the practical choice for real-time robotic MEP monitoring remains a fine-tuned lightweight detector.

Load-bearing premise

The comparison assumes that running each open-vocabulary model with its default settings and no special wording is a fair test of what it can do for MEP detection.

Editorial extensions

If this is right

  • For construction-site MEP monitoring with mobile robots, the results imply that a fine-tuned lightweight detector is the currently reliable option, at least until open-vocabulary models are adapted to the domain.
  • Because YOLO11 Nano runs at 23.36 FPS on an NVIDIA Jetson Orin Nano, the approach fits the compute budget of an onboard robot system, whereas the largest open-vocabulary model (910M parameters) is far harder to deploy on such hardware.
  • The open-vocabulary models' near-zero scores on classes like cable tray fittings indicate that generic pretraining data (COCO, LVIS) does not transfer to MEP components, so gains will likely require domain-specific data rather than larger generic models alone.
  • On the dataset released with the paper, future open-vocabulary models can be measured against these numbers, giving the community a concrete baseline to beat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gap is specifically against zero-shot, default-configuration use of the open-vocabulary models; the paper does not report prompt texts or confidence thresholds, so a prompt-engineered or lightly fine-tuned version of these models could plausibly narrow the gap on the same test set.
  • If that happened, the practical takeaway would shift toward a hybrid: a fast fine-tuned detector for known MEP classes plus an open-vocabulary model as a fallback for novel or rare components.
  • Because the dataset is shared, one can track future vision-language models against the same 10 classes; a reasonable target would be matching YOLO11's F1 without per-class fine-tuning, which would amount to a definition of 'ready for MEP detection.'
  • The single-site data collection leaves open whether the fine-tuned model's advantage persists across different construction sites, lighting conditions, and camera viewpoints; that cross-site generalization question is not addressed by the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper evaluates whether open-vocabulary detection models are ready for detecting MEP (mechanical, electrical, plumbing) components on construction sites. The authors collected a real-world dataset using a mobile ground robot, annotated 10 MEP component classes, and compared a fine-tuned YOLO11 Nano detector against three pre-trained open-vocabulary models (Grounding SAM2, Grounding DINO, and DETIC) used without fine-tuning. The reported results show a very large performance gap: YOLO11 Nano achieves an overall F1 of 0.89, while the open-vocabulary models achieve F1 scores between 0.014 and 0.032 on the same test split. The paper concludes that fine-tuned lightweight models still largely outperform open-vocabulary models in specialized environments and domain-specific tasks.

Significance. If the comparison is taken at face value, the paper provides a useful data point for practitioners in construction robotics: for this specific MEP detection task, a 2.6M-parameter fine-tuned detector massively outperforms much larger zero-shot open-vocabulary models. The main strength is the independently collected, annotated, and publicly shared dataset, which is a valuable resource for future work. The paper also reports deployment-relevant metrics (model size, inference speed on an embedded platform). However, the central quantitative claim rests on an under-specified evaluation protocol: the open-vocabulary models are used zero-shot with undisclosed prompts and confidence thresholds, no error bars are reported, and no attempt is made to tune or adapt the open-vocabulary models. The qualitative direction of the result is likely robust given the size of the gap, but the exact F1 values and the broad 'not ready' framing are conditional on missing details.

major comments (3)
  1. [Section 2, Evaluation Procedure; Table 2] The evaluation procedure does not report the text prompts, class vocabularies, or detection confidence thresholds used for Grounding SAM2, Grounding DINO, and DETIC. For open-vocabulary detectors these choices strongly control the precision-recall trade-off and can change reported F1 by orders of magnitude. Without this information, the near-zero F1 values in Table 2 cannot be interpreted as a fair representation of what these models can achieve for MEP detection. The manuscript should specify the exact prompt strings (e.g., 'pipe fitting', 'cable tray fitting', or a combined prompt), the box/text thresholds, and any post-processing filters. If the authors used generic prompts or default thresholds, the conclusion should be narrowed to 'default, untuned open-vocabulary models' rather than 'open-vocabulary models are not ready'.
  2. [Section 3, Results and Discussion; Table 2] No error bars, repeated runs, or statistical significance tests are reported. The test split contains relatively few instances for some classes (e.g., Generator has 124 instances, Cable Tray Fitting has 317), so per-class F1 values such as 0.0 vs 0.005 may not be meaningful. Given the small class sizes and the single train/validation/test split, the authors should provide confidence intervals or at least report the number of true positives/false positives underlying the aggregate F1 scores.
  3. [Section 4, Conclusion] The conclusion states that fine-tuned models 'significantly outperformed open-vocabulary models in detection accuracy, computational efficiency, and real-time applicability' and that open-vocabulary performance 'remains limited.' This overgeneralizes because the open-vocabulary models were only evaluated zero-shot with undisclosed prompts and thresholds, and no prompt engineering, prompt tuning, or fine-tuning was attempted. The paper's own future-work suggestions (fine-tuning and construction-related prompts) indicate that the evidence supports a narrower claim: a fine-tuned lightweight detector outperforms untuned open-vocabulary models in this specific setup. The abstract and conclusion should be revised to reflect this scope.
minor comments (4)
  1. [Section 2, Dataset] The statement 'The dataset is shared online' does not include a link or repository identifier; the reader cannot access the data without further searching.
  2. [Section 3, Real-world Applicability] The manuscript reports inference speed only for YOLO11 Nano (23.36 FPS on Jetson Orin Nano) and does not report inference times for the three open-vocabulary models on the same hardware. The claim that open-vocabulary models are 'less practical' for real-time use would be stronger with at least approximate runtime comparisons.
  3. [Section 3, Qualitative examples] The sentence 'the model outperformed human annotations' is not supported by a quantitative measure; it may be more precise to say that the model detected objects that the annotators had missed, which is a qualitative observation.
  4. [Table 1] The row labeled 'All' reports total images as 8885, but the sum of per-class image counts exceeds this number because images contain multiple classes; this is fine, but the table caption should clarify that 'Images' counts are not disjoint across classes.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark on an independently collected dataset, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's central claim is that a fine-tuned, lightweight YOLO11 Nano outperforms three pre-trained open-vocabulary detectors on a newly collected and manually annotated MEP dataset. This is an externally grounded empirical comparison, not a derivation from assumptions that encode the conclusion. The fine-tuned model is trained on one split of the dataset and evaluated on a held-out test split; the open-vocabulary models are evaluated zero-shot on the same test split. No equation in the paper reduces the outcome to its inputs, and no fitted parameter is renamed as a prediction. The authors' self-citations, such as reference [2] on robotic deployment considerations, are contextual and not load-bearing for the quantitative comparison. Concerns about undisclosed prompts or confidence thresholds for the open-vocabulary models are evaluation-validity issues, not circularity: they question whether the baseline was configured optimally, but they do not show that any result is true by construction. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical comparison, so there are no fitted physical constants or invented theoretical entities. The main hidden inputs are reporting choices: unstated prompt templates and confidence thresholds for the open-vocabulary models, plus the assumption that one site's annotations generalize to MEP detection broadly.

free parameters (2)
  • Detection confidence thresholds for open-vocabulary models = not reported (likely defaults)
    F1 values for GSAM2, GDINO, and DETIC depend critically on confidence threshold. Without reporting the thresholds, the reported near-zero F1 values could understate the models' capabilities and cannot be reproduced.
  • Prompt templates for open-vocabulary models = not reported
    Open-vocabulary detection results depend strongly on prompt wording. The paper does not specify the prompts used for any of the three open-vocabulary models, so the comparison may not reflect their best zero-shot performance.
assumptions (3)
  • domain assumption The manually annotated dataset is representative of MEP components on active construction sites and the labels are correct.
    Data was collected at one industrial partner's site and annotated manually, but no annotation quality metrics, inter-annotator agreement, or QA process is reported (Dataset paragraph, Section 2).
  • domain assumption The 10% test split from a single construction site supports general conclusions about open-vocabulary model readiness for MEP detection.
    The paper generalizes beyond the specific site and robot, but no cross-site or cross-robot evaluation is included. This assumption enters in the conclusion and real-world applicability discussion.
  • domain assumption Zero-shot evaluation with no prompt tuning or threshold calibration fairly represents the capability of open-vocabulary models for this task.
    The evaluation procedure compares fine-tuned YOLO11 against pre-trained open-vocabulary models without fine-tuning. If prompts or thresholds were tuned, the gap might narrow, so this assumption is load-bearing for the broad conclusion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites." pith.science (2026). https://pith.science/paper/64RGUMPA

@misc{pith2026250109267,
  author       = {Pith},
  title        = {Pith review of: Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64RGUMPA}},
  note         = {Machine review of arXiv:2501.09267}
}
read the original abstract

The construction industry has long explored robotics and computer vision, yet their deployment on construction sites remains very limited. These technologies have the potential to revolutionize traditional workflows by enhancing accuracy, efficiency, and safety in construction management. Ground robots equipped with advanced vision systems could automate tasks such as monitoring mechanical, electrical, and plumbing (MEP) systems. The present research evaluates the applicability of open-vocabulary vision-language models compared to fine-tuned, lightweight, closed-set object detectors for detecting MEP components using a mobile ground robotic platform. A dataset collected with cameras mounted on a ground robot was manually annotated and analyzed to compare model performance. The results demonstrate that, despite the versatility of vision-language models, fine-tuned lightweight models still largely outperform them in specialized environments and for domain-specific tasks.

Figures

Figures reproduced from arXiv: 2501.09267 by the authors.

Figure 1
Figure 1. Our Journeybot platform for data collec [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. provides qualitative examples of YOLO11 Nano’s predictions, displaying predicted classes, bound￾ing boxes, and confidence scores. In some cases, the model outperformed human annotations, identifying over￾looked objects like valves in the bottom-right corner of an image. Conversely, in instances of blurred images, the model avoided false positives, aligning with human anno￾tators’ limitations. However, false detectio… view at source ↗
Figure 2
Figure 2. presents the normalized confusion matrix for YOLO11 Nano’s predictions, demonstrating robust per￾formance even for less-represented classes, such as gener￾ators, with a detection accuracy of 91%. This emphasizes YOLO11’s capability to handle imbalanced datasets, a common challenge in domain-specific applications [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

  1. [1]

    Robotics in the construction industry: State of the art and future opportunities

    Guglielmo Carra, Alfredo Argiolas, Alessandro Bel- lissima, Marta Niccolini, and Matteo Ragaglia. Robotics in the construction industry: State of the art and future opportunities. In ISARC. Proceed- ings of the International Symposium on Automation and Robotics in Construction, volume 35, pages 1–8. IAARC Publications, 2018

  2. [2]

    Robotic deployment on construction sites: considerations for safety and productivity impact

    Rafael Gomes Braga, Muhammad Owais Tahir, Ivanka Iordanova, and David St-Onge. Robotic deployment on construction sites: considerations for safety and productivity impact. arXiv preprint arXiv:2404.13143, 2024

  3. [3]

    Future prospects of the assembly model for mep systems in chinese buildings: A whole life cycle perspective

    Chun Wang, Peng Ouyang, Xiaodong Liu, Zhihua Zou, Yuanping Wang, and Ruiling Wang. Future prospects of the assembly model for mep systems in chinese buildings: A whole life cycle perspective. Applied Sciences, 14(15):6818, 2024

  4. [4]

    An object-based 3d walk-through model for interior construction progress monitoring

    Seungjun Roh, Zeeshan Aziz, and Feniosky Pena- Mora. An object-based 3d walk-through model for interior construction progress monitoring. Automa- tion in Construction, 20(1):66–75, 2011

  5. [5]

    Small and overlapping worker detection at construc- tion sites

    Minsoo Park, Jinyeong Bak, Seunghee Park, et al. Small and overlapping worker detection at construc- tion sites. Automation in Construction, 151:104856, 2023

  6. [6]

    Casnli-yolo: construction site multi-target detection method based on improved yolov5s

    Fei Hao, Tianqi Zhang, Guanghong He, Ruofei Dou, and Chao Meng. Casnli-yolo: construction site multi-target detection method based on improved yolov5s. Measurement Science and Technology , 35 (8):085202, 2024

  7. [7]

    Object detection using yolo: Challenges, architectural successors, datasets and applications

    Tausif Diwan, G Anirudh, and Jitendra V Temb- hurne. Object detection using yolo: Challenges, architectural successors, datasets and applications. multimedia Tools and Applications , 82(6):9243– 9275, 2023

  8. [8]

    Yolo-world: Real- time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 16901–16911, 2024

Show all 17 references
  1. [9]

    Autorepo: A general framework for multimodal llm-based automated construction reporting

    Hongxu Pu, Xincong Yang, Jing Li, and Runhao Guo. Autorepo: A general framework for multimodal llm-based automated construction reporting. Expert Systems with Applications, 255:124601, 2024

  2. [10]

    Omni-scan2bim: A ready-to-use scan2bim approach based on vision foundation models for mep scenes

    Boyu Wang, Zhengyi Chen, Mingkai Li, Qian Wang, Chao Yin, and Jack CP Cheng. Omni-scan2bim: A ready-to-use scan2bim approach based on vision foundation models for mep scenes. Automation in Construction, 162:105384, 2024

  3. [11]

    Constscene: Dataset and model for advancing robust semantic segmentation in construction environments

    Maghsood Salimi, Mohammad Loni, Sara Afshar, Antonio Cicchetti, and Marjan Sirjani. Constscene: Dataset and model for advancing robust semantic segmentation in construction environments. arXiv preprint arXiv:2312.16516, 2023

  4. [12]

    Yolov11: An overview of the key architectural enhancements

    R Khanam and M Hussain. Yolov11: An overview of the key architectural enhancements. arxiv 2024. arXiv preprint arXiv:2410.17725

  5. [13]

    Grounded sam: As- sembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: As- sembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024

  6. [14]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jian- wei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vi- sion, pages 38–55. Spri...

  7. [15]

    Detecting twenty- thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨ uhl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference on Computer Vision , pages 350–368. Springer, 2022

  8. [16]

    Microsoft coco: Com- mon objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proce...

  9. [17]

    LVIS: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.