REVIEW 4 major objections 5 minor 29 references
Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Zenesis combines text-prompted GroundingDINO boxes with SAM masks to segment noisy FIB-SEM catalyst images at 0.858 IoU and 0.923 Dice without retraining.
desk verdict A useful no-code wrapper for GroundingDINO+SAM, but the headline numbers rest on an unexplained ground-truth source that the paper itself says doesn't exist. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the GroundingDINO-to-SAM cascade. A Swin-T GroundingDINO model projects text and image into a shared embedding space and outputs bounding boxes from cross-modal attention; the ViT-H Segment Anything Model then uses each box as a prompt to produce a pixel-level mask, and the paper names this combined system Zenesis. Two correction mechanisms support it: a human-in-the-loop 'Rectify Segmentation' step that lets users replace outlying boxes with user-selected ones, and a sliding-window heuristic that replaces boxes exceeding a height or width factor with the average box of neighboring slices to keep volumetric segmentations temporally consistent.
What would settle it
Recompute the reported IoU and Dice using independently produced expert masks on the same 20 FIB-SEM slices, with at least two annotators and a reported inter-annotator agreement. If independent masks yield overlap substantially below 0.86 IoU, or if the original evaluation masks turn out to be derived from the system's own heuristic refinement, the central accuracy claim would not hold.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that zero-shot segmentation of raw scientific images is achievable by grounding a promptable mask generator in text-conditioned detection. The combined pipeline, Zenesis, reaches average IoU of 0.858 and Dice of 0.923 on amorphous FIB-SEM catalyst layers and 0.857 IoU and 0.923 Dice on crystalline ones, with accuracies of 0.947 and 0.987 respectively. The paper attributes the gain to GroundingDINO's text-prompted boxes keeping SAM from following the sharp background gradient that derails Otsu thresholding and SAM-only, which score as low as 0.100 IoU on crystalline slices. It frames this as a data-readiness contribution: raw, unannotated, noisy scientific data can be segmented directly, and only a small labeled set is needed for evaluation.
Load-bearing premise
The accuracy claim assumes trustworthy ground-truth masks exist for the 20 FIB-SEM slices, but the Dataset Description says the dataset 'lack(s) of ground-truth annotations' and no passage explains how the evaluation masks were produced or by whom.
Editorial extensions
If this is right
- On crystalline FIB-SEM slices, text-grounded boxes lift IoU from 0.100 with SAM-only and 0.161 with Otsu thresholding to 0.857, removing the failure mode of latching onto the black background gradient.
- Because segmentation needs only a natural-language prompt and no training data, a domain expert without programming or ML experience can process a raw image or volume directly, shortening the data-readiness step.
- The rectification and temporal-averaging heuristics give volumetric consistency, so an anomalous detection in one slice can be corrected rather than propagated through the stack.
- The reported CPU timings of 8 to 10 seconds across image sizes from 640 by 480 to 2560 by 2560 indicate interactive use on consumer hardware, and the paper argues a GPU-accelerated SAM2 version would reach sub-second per-slice processing.
- The paper reports preliminary evaluations on cryo-TEM and micro-CT images with comparable or larger improvements, which would extend the cascade beyond FIB-SEM.
Reading between the lines
- The reported IoU and Dice numbers should be read as conditional on reference masks whose origin is not described; until those masks are released or reproduced by an independent annotation protocol, the metrics demonstrate pipeline behavior on this dataset rather than a validated benchmark.
- A natural ablation the paper does not report would compare Zenesis against SAM prompted with the same GroundingDINO boxes but no text, or against boxes from an unrelated detector; that would isolate how much of the gain comes from text grounding versus from box prompting itself.
- Because the heuristic refinement replaces anomalous boxes with the average of neighboring slices, it imposes a smoothness prior that could suppress genuinely abrupt structural changes; a sensitivity sweep over the replacement factor would reveal where the heuristic helps and where it hides real features.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Zenesis, an interactive no-code platform that chains GroundingDINO text-conditioned bounding-box detection with SAM mask generation to segment scientific images without AI-ready annotations. The authors validate the system on 20 FIB-SEM slices of amorphous and crystalline catalyst layers, reporting average IoU/Dice of 0.858/0.923 and 0.857/0.923, respectively, and claim substantial gains over Otsu thresholding and SAM-only baselines. The paper also describes human-in-the-loop box correction, heuristic volumetric refinement, and a real-time evaluation dashboard.
Significance. If the reported results are trustworthy, Zenesis would be a practically useful bridge between raw scientific imaging data and off-the-shelf foundation models, and the paper would provide a useful systems contribution for materials-science audiences. The approach has a genuine strength: it uses pretrained GroundingDINO and SAM without parameter fitting to the test set, so the reported gains are not an artifact of test-set training. However, the significance is currently conditional on evaluation details that are missing: the dataset is explicitly described as lacking ground-truth annotations, yet IoU/Dice scores are reported without any account of how reference masks were produced. Without that provenance, and without code or data release, the central quantitative claim is not verifiable. The paper's framing of 'zero-shot' performance is also muddled by the presence of interactive correction and heuristic replacement steps whose role in the reported numbers is never stated.
major comments (4)
- [Dataset Description] The Dataset Description states that the dataset presents 'the lack of ground-truth annotations', yet Table 1 reports IoU, Dice, and accuracy against reference masks for 20 slices. The paper never explains who created the evaluation masks, by what procedure, under what instructions, or whether the masks were validated independently. This is load-bearing because every headline number in the paper is a comparison to those masks. Please specify the ground-truth generation protocol, the number of annotators, inter-annotator agreement, and the criteria used to segment the catalyst layer.
- [Results and Analysis / Table 1] The Zenesis row in Table 1 likely includes more than the automatic GroundingDINO+SAM pipeline, because the method description includes 'Rectify Segmentation' human-in-the-loop box adjustments and 'Automated Heuristic Refinement' that replaces outlier boxes with averaged boxes from neighboring slices. The paper does not state whether these components were used when producing the Table 1 numbers, nor what fraction of slices received human correction. If human corrections were applied before scoring, the result is not a zero-shot evaluation; if they were not applied, the description of the evaluated system is incomplete. Please state explicitly which pipeline variant produced each row of Table 1.
- [Theoretical Framework and Workflow] The pipeline has several free parameters that are never reported: the GroundingDINO box_threshold and text_threshold, and the heuristic box size factor used in the volumetric refinement. Since the paper claims that GroundingDINO guidance is 'pivotal' for the performance gain, a sensitivity analysis over these thresholds is needed to establish that the reported 0.858/0.857 IoU values are robust rather than a favorable threshold choice. At minimum, report the exact values used for all experiments.
- [Results and Analysis] The experimental comparison does not isolate the contribution of the text-grounded bounding boxes. The baselines are only Otsu and SAM-only, so it is unclear how much of the gain comes from GroundingDINO versus from SAM with a reasonable bounding box, or from the heuristic refinement. An ablation with GroundingDINO-only masks, SAM with oracle/ground-truth boxes, and Zenesis without the volumetric heuristic would make the central claim—that text grounding is pivotal—actually testable. Without such an ablation, the paper overstates what can be concluded from the three rows of Table 1.
minor comments (5)
- [Abstract and Results and Analysis] The metric 'accuracy' is used prominently in the abstract and Table 1 but never defined. Please specify whether this is pixel accuracy, and how it is computed relative to foreground/background imbalance.
- [Results and Analysis] The paper reports timings for three images of different sizes but does not state the number of repeats, the text prompts used, or whether the measurements include model loading time. Please provide a precise timing protocol.
- [Related Work] The mention of 'SAM 2' in the foundation-model background is not connected to the experiments, which use 'SAM1' only. Please clarify whether SAM 2 is evaluated or only mentioned as future work.
- [Theoretical Framework and Workflow] The attention formula is standard but is presented without stating which matrices come from the text branch and which from the image branch. A sentence clarifying the notation and the role of the cross-modal projection would improve readability.
- [Dataset Description] The paper uses 10 slices per class from a 3D volume but does not state whether the slices are contiguous or selected to cover different regions; this affects how representative the average metrics are of volumetric performance. Please specify the slice selection strategy.
Circularity Check
No circular derivation: Zenesis applies pre-trained GroundingDINO and SAM with no test-set fitting; reported IoU/Dice depend on an undescribed ground-truth source, a data-provenance caveat rather than a circular argument.
full rationale
The claimed derivation chain is an engineering pipeline, not a mathematical reduction. GroundingDINO produces bounding boxes from text prompts via standard cross-modal attention; SAM converts boxes into masks; heuristic box replacement uses adjacent-slice averages. No parameter is fitted to the 20 FIB-SEM test slices and no result is defined in terms of another claimed output. The only self-references are to prior membrane-electrode work ([7,14]) and are background, not load-bearing. The dataset description states the data has 'lack of ground-truth annotations' while Table 1 reports IoU/Dice against ground truth; the paper never says who made the reference masks or whether the interactive 'Rectify Segmentation' corrections influenced the scored outputs. That is an evaluation-provenance concern that could make the numbers optimistic, but it is not a case where a prediction is equivalent by construction to an input, a fitted parameter is renamed a prediction, or a self-citation forces the conclusion. Therefore the circularity score is minimal.
Assumptions & free parameters
free parameters (3)
- GroundingDINO box_threshold
- GroundingDINO text_threshold
- Heuristic box size factor
assumptions (4)
- domain assumption Pre-trained GroundingDINO and SAM transfer to grayscale scientific images without fine-tuning
- domain assumption The text prompt 'catalyst layer' (or similar) is sufficient for DINO to locate the target in all slices
- domain assumption The 20 selected slices are representative of the two volumes
- ad hoc to paper Evaluation ground-truth masks are accurate despite the dataset being described as lacking annotations
Cite this review
Pith. "Pith review of Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data." pith.science (2026). https://pith.science/paper/YIS6AIHM
@misc{pith2026250624039,
author = {Pith},
title = {Pith review of: Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/YIS6AIHM}},
note = {Machine review of arXiv:2506.24039}
}
read the original abstract
Zero-shot and prompt-based models have excelled at visual reasoning tasks by leveraging large-scale natural image corpora, but they often fail on sparse and domain-specific scientific image data. We introduce Zenesis, a no-code interactive computer vision platform designed to reduce data readiness bottlenecks in scientific imaging workflows. Zenesis integrates lightweight multimodal adaptation for zero-shot inference on raw scientific data, human-in-the-loop refinement, and heuristic-based temporal enhancement. We validate our approach on Focused Ion Beam Scanning Electron Microscopy (FIB-SEM) datasets of catalyst-loaded membranes. Zenesis outperforms baselines, achieving an average accuracy of 0.947, Intersection over Union (IoU) of 0.858, and Dice score of 0.923 on amorphous catalyst samples; and 0.987 accuracy, 0.857 IoU, and 0.923 Dice on crystalline samples. These results represent a significant performance gain over conventional methods such as Otsu thresholding and standalone models like the Segment Anything Model (SAM). Zenesis enables effective image segmentation in domains where annotated datasets are limited, offering a scalable solution for scientific discovery.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Segment anything for microscopy
Anwai Archit, Luca Freckmann, Sushmita Nair, Nabeel Khalid, Paul Hilt, Vikas Rajashekar, Marei Freitag, Carolin Teuber, Melanie Spitzner, Con- stanza Tapia Contreras, et al. Segment anything for microscopy. Nature Methods, 22(3):579–591, 2025
work page 2025
-
[2]
Emerging properties in self- supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 9650–9660, 2021
work page 2021
-
[3]
Anne E. Carpenter, Thouis R. Jones, Michael R. Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David A. Guertin, Joo Han Chang, Robert A. Lindquist, Jason Moffat, Polina Golland, and David M. Sabatini. Cellprofiler: image analysis software for identifying and quantifying cell pheno- types. Genome Biology, 7(10):R100, 2006
work page 2006
-
[4]
Lienkamp, Thomas Brox, and Olaf Ronneberger
¨Ozg¨ un C ¸ i¸ cek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. In International Confer- ence on Medical Image Computing and Computer- Assisted Intervention , pages 424–432. Springer, 2016
work page 2016
-
[5]
Measures of the amount of ecologic as- sociation between species
Lee R Dice. Measures of the amount of ecologic as- sociation between species. Ecology, 26(3):297–302, 1945
work page 1945
-
[6]
The pascal visual object classes (voc) challenge
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inter- national journal of computer vision, 88(2):303–338, 2010
work page 2010
-
[7]
Achieving the hydrogen shot: Interrogating ionomer interfaces
Julie C Fornaciari, Shannon Boettcher, Ethan Crumlin, Ahmet Kusoglu, David Prendergast, Daniela Ushizima, Iryna Zenyuk, and Adam Z We- ber. Achieving the hydrogen shot: Interrogating ionomer interfaces. MRS Energy & Sustainability, 12(1):14–22, 2025
work page 2025
-
[8]
Deep learning analysis on microscopic imag- ing in materials science
Mengshu Ge, Fei Su, Zhicheng Zhao, and Dong Su. Deep learning analysis on microscopic imag- ing in materials science. Materials Today Nano, 11:100087, 2020
work page 2020
Show all 29 references
-
[9]
VISTA3D: A Unified Segmenta- tion Foundation Model For 3D Medical Imaging
Yufan He, Pengfei Guo, Yucheng Tang, Andriy Myronenko, Vishwesh Nath, Ziyue Xu, Dong Yang, Can Zhao, Benjamin Simon, Mason Belue, Stephanie Harmon, Baris Turkbey, Daguang Xu, and Wenqi Li. VISTA3D: A Unified Segmenta- tion Foundation Model For 3D Medical Imaging. In Proceeding...
2025
-
[10]
Data readiness for ai: A 360-degree survey
Kaveen Hiniduma, Suren Byna, and Jean Luca Bez. Data readiness for ai: A 360-degree survey. ACM Computing Surveys, 57(9):1–39, 2025
2025
-
[11]
Referitgame: Referring to objects in photographs of natural scenes
Sahar Kazemzadeh, Vicente Ordonez, Mark Mat- ten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Pro- ceedings of the 2014 conference on empirical meth- ods in natural language processing (EMNLP), pages 787–798, 2014
2014
-
[12]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on com- puter vision, pages 4015–4026, 2023
2023
-
[13]
Computer visualization of three-dimensional image data using imod
James R Kremer, David N Mastronarde, and J Richard McIntosh. Computer visualization of three-dimensional image data using imod. Journal of structural biology, 116(1):71–76, 1996
1996
-
[14]
Understanding structure differences of irid- ium oxides depends on loading in proton exchange membrane electrolyzers
Obeen Kwon, Florian Chabot, Carol Korzeniewski, Shannon Boettcher, Yu Morimoto, and Iryna Zenyuk. Understanding structure differences of irid- ium oxides depends on loading in proton exchange membrane electrolyzers. In Electrochemical Soci- ety Meeting Abstracts prime2024, num...
2024
-
[15]
Brickdl: Graph- level optimizations for dnns with fine-grained data blocking on gpus
Mahesh Lakshminarasimhan, Mary Hall, Samuel Williams, and Oscar Antepara. Brickdl: Graph- level optimizations for dnns with fine-grained data blocking on gpus. In Proceedings of the 53rd Inter- national Conference on Parallel Processing, pages 576–586, 2024
2024
-
[16]
Grounding dino: Mar- rying dino with grounded pre-training for open-set object detection
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Mar- rying dino with grounded pre-training for open-set object detection. In European conference on com- puter vision, pages 38–55. Spri...
2024
-
[17]
Image seg- mentation using text and image prompts
Timo L¨ uddecke and Alexander Ecker. Image seg- mentation using text and image prompts. arXiv preprint arXiv:2112.10003, 2022
2022 arXiv
-
[18]
Segment anything in medical images
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):1–9, 2024
2024
-
[19]
Mazurowski, Haoyu Dong, Han Gu, Jichen Yang, Nicholas Saha, and Jiarui Luo
Maciej A. Mazurowski, Haoyu Dong, Han Gu, Jichen Yang, Nicholas Saha, and Jiarui Luo. Seg- ment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89:102918, 2023. 8
2023
-
[20]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Robin R¨ ombach, Chloe Rol- land, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[21]
Grounded sam: As- sembling open-world models for diverse visual tasks
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: As- sembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024
2024 arXiv
-
[22]
Nih image to imagej: 25 years of image analysis
Caroline A Schneider, Wayne S Rasband, and Kevin W Eliceiri. Nih image to imagej: 25 years of image analysis. Nature methods, 9(7):671–675, 2012
2012
-
[23]
Parameter-efficient quantized mixture-of- experts meets vision-language instruction tuning for semiconductor electron micrograph analysis
Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, and Venkataramana Runk- ana. Parameter-efficient quantized mixture-of- experts meets vision-language instruction tuning for semiconductor electron micrograph analysis. arXiv preprint arXiv:2408.15305, 2024
2024 arXiv
-
[24]
Interactive medical image segmentation using deep learning with image-specific fine tuning
Guotai Wang, Wenqi Li, Maria A Zuluaga, Rosalind Pratt, Premal A Patel, Michael Aertsen, Tom Doel, Anna L David, Jan Deprest, S´ ebastien Ourselin, et al. Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE transactions on medical i...
2018
-
[25]
Deepigeos: a deep interactive geodesic frame- work for medical image segmentation
Guotai Wang, Maria A Zuluaga, Wenqi Li, Rosalind Pratt, Premal A Patel, Michael Aertsen, Tom Doel, Anna L David, Jan Deprest, S´ ebastien Ourselin, et al. Deepigeos: a deep interactive geodesic frame- work for medical image segmentation. IEEE trans- actions on pattern analysis...
2018
-
[26]
Ushizima, Victor C
Ke Xu, Qingxu Jin, Jiaqi Li, Daniela M. Ushizima, Victor C. Li, Kimberly E. Kurtis, and Paulo J.M. Monteiro. In-situ microtomography image segmen- tation for characterizing strain-hardening cementi- tious composites under tension using machine learn- ing. Cement and Concrete R...
2023
-
[27]
Faster segment anything: Towards lightweight sam for mobile applications
Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023
2023 arXiv
-
[28]
Fast segment anything
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306.12156, 2023
2023 arXiv
-
[29]
Meduhip: Towards human-in-the-loop medical segmentation
Jiayuan Zhu and Junde Wu. Meduhip: Towards human-in-the-loop medical segmentation. arXiv preprint arXiv:2408.01620, 2024. 9
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.