REVIEW 3 major objections 4 minor 40 references
Leveraging Computational Pathology AI for Noninvasive Optical Imaging Analysis Without Retraining
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Converting OCT and RCM scans into H&E-like images lets off-the-shelf pathology AI analyze them accurately without retraining.
desk verdict The OCT half is a solid, citable result; the RCM cell-segmentation claim is weakened by a circular ground truth and needs an independent re-annotation before the paper's strongest claims stand. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a domain transfer network (also called virtual staining), such as OCT2Hist for OCT and the Li et al. model for RCM, which maps an optical image into an H&E-like image. These networks are trained on co-registered optical and H&E image pairs, typically using a few thousand images and micrometer-level registration accuracy. The H&E-like image is then passed to an off-the-shelf computational pathology model (SAM, MedSAM, SAM-Med2D, Hover-Net, PLIP, UNI, or GPT-4o) in zero-shot mode. The Kullback-Leibler (KL) divergence between virtual H&E and real H&E embeddings, computed with VGG16, is the explanatory metric: reducing the distance to the training domain is what the paper credits for the accuracy gains.
What would settle it
Obtain an independently annotated RCM cell segmentation ground truth, for example manual tracing by a dermatopathologist on raw RCM or a much larger CellProfiler-derived set with manual correction, then re-run the comparison on the same images or a larger cohort. If domain transfer plus Hover-Net no longer outperforms CellProfiler, or the gap narrows substantially, the central RCM cell segmentation claim fails. For the broader claim, apply FoundationShift to a noninvasive modality whose virtual staining model is known to be less accurate and check whether accuracy gains disappear as KL divergence to H&E increases.
Extended reading notes
Core claim
FoundationShift's central claim is that converting optical images (OCT and RCM) to H&E-like images before using off-the-shelf computational pathology models significantly improves model accuracy. The paper demonstrates this across multiple models: SAM, MedSAM, and SAM-Med2D for epidermis segmentation in OCT (Dice scores rise from 0.68, 0.67, and 0.38 to 0.73, 0.80, and 0.74, respectively); Hover-Net for RCM cell segmentation, where raw Hover-Net segments zero cells but domain-transferred Hover-Net segments 4,503 cells and outperforms CellProfiler; PLIP for OCT image retrieval (recall@5 rises from 19% to 93%); GPT-4o for tissue identification (from 0% correct to over 80%); and UNI for BCC nodule classification in RCM (F1 score rises from 0.35 to 0.85). The authors also show that virtual H&E images reduce KL divergence to the H&E domain more than a comparable CT-to-MR domain transfer does.
Load-bearing premise
The RCM cell segmentation ground truth was created by running Hover-Net on the virtual H&E images and manually adjusting the results; if that ground truth is biased toward Hover-Net, the reported superiority over CellProfiler would not be supported.
Editorial extensions
If this is right
- Domain transfer plus MedSAM yields a statistically significant Dice improvement on OCT epidermis segmentation ($p < 2 \times 10^{-15}$), with the largest gains for models that were trained on pathology images.
- FoundationShift produces the first zero-shot cell segmentation pipeline for RCM images, since Hover-Net alone segments no cells on raw RCM but works on virtual H&E.
- PLIP, a visual-language model with no OCT training, reaches 93% recall@5 on OCT after domain transfer, comparable to the 82% achieved on real H&E images.
- GPT-4o identifies the tissue region in over 80% of virtual H&E images versus 0% on raw OCT, making LLM-based description of noninvasive scans feasible.
- Because OCT and RCM are 3D modalities, FoundationShift extends 2D pathology foundation models to 3D volumes, demonstrated by a 3D epidermis and BCC nodule segmentation.
Reading between the lines
- If FoundationShift generalizes to other virtual staining models (photoacoustic, Raman, two-photon), a single pathology foundation model could serve many noninvasive modalities without retraining, potentially changing deployment of optical imaging AI in dermatology and beyond.
- The reported KL-divergence reduction predicts that models whose training domain is closest to H&E will benefit most; a testable extension is to vary domain-transfer fidelity and measure accuracy versus KL distance across modalities.
- The RCM cell segmentation ground truth was seeded by Hover-Net on virtual H&E, so the reported margin over CellProfiler may partly reflect this construction; an independent manual ground truth or a substantially larger CellProfiler-based control would settle the unbiasedness question.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FoundationShift, a method for applying off-the-shelf computational pathology (CPath) models to noninvasive optical images by first translating OCT and RCM images into H&E-like 'virtual histology' images using existing domain-transfer models, without retraining or fine-tuning the CPath models. The authors evaluate FoundationShift on several tasks: epidermis segmentation in OCT using SAM, MedSAM, and SAM-Med2D; cell segmentation in RCM using Hover-Net compared with CellProfiler; image retrieval using PLIP; BCC nodule classification using UNI; and image captioning using GPT-4o. The central claim is that domain transfer to H&E-like images significantly improves the accuracy of these models on noninvasive optical images, thereby enabling zero-shot use of CPath foundation models on OCT and RCM data.
Significance. The OCT segmentation results are the strongest part of the paper: they use 95 paired OCT/H&E images, expert ground truth with inter-observer agreement analysis, paired Wilcoxon tests, and consistent improvements across three segmentation models, with data and code links provided. If the RCM cell-segmentation result were independently validated, FoundationShift would be a practical and potentially impactful way to repurpose the large ecosystem of H&E-trained CPath models for noninvasive optical imaging, which is a clinically relevant direction. The paper is also commendable for evaluating multiple model families (generic segmentation, medical segmentation, cell segmentation, vision-language models, large language models) and for explicitly discussing limitations of the domain-transfer step. However, the RCM cell-segmentation evaluation is entangled with its own ground truth, and several secondary claims (UNI classification, ChatGPT analysis) rest on small or subjective evaluations. The overall significance is high if the RCM circularity can be resolved, but current evidence is mixed.
major comments (3)
- [Methods, RCM Cell Segmentation and Statistical Analysis; Figure 2c] The RCM cell-segmentation ground truth was created by first running Hover-Net on the virtually stained H&E images and then manually adjusting the results, as stated in this section. Because the evaluated method is exactly Hover-Net applied to the same virtual H&E input, this creates a circular evaluation: the model's own outputs seed the reference standard. The authors acknowledge that this 'may skew results in favor of Hover-Net,' but the 50-cell control using a CellProfiler-derived ground truth is insufficient: 50 cells is roughly 1% of the 4,503-cell evaluation set, the control still involves manual adjustment within the same workflow, and the reported Dice score of 0.9 between the two ground truths does not directly bound the DQ/SQ/PQ differences used for the headline comparison. This circularity is load-bearing because RCM is one of only two modalities supporting the paper's main claim, and the RCM experiment is the only quantitative cell-level result. The authors should provide an independently annotated ground truth, for example manual expert annotation on raw RCM images or on virtual H&E with an annotation protocol that does not involve Hover-Net, and re-run the comparison before claiming that FoundationShift 'significantly enhances the accuracy of ... Hover-Net' for RCM cell segmentation.
- [Methods, OCT Tissue Segmentation and Statistical Analysis] The bounding box prompt used for all OCT segmentation models is derived from expert annotations, with a height factor adjustment of 1.15, as described in this section. Since the same expert annotations define the ground truth used for Dice evaluation, the segmentation models receive prompt information that is already aligned with the reference standard. This is a form of ground-truth leakage that inflates absolute Dice scores and makes the reported accuracies unrepresentative of a true zero-shot deployment, where users would need to supply or generate prompts without access to expert ground truth. The relative improvement from domain transfer is likely still informative because the same prompts are used in both conditions, but the paper's claim that FoundationShift 'significantly improves model accuracy' for off-the-shelf models needs to be qualified by this protocol. Please state clearly that the evaluation assumes expert-provided bounding boxes, and, if feasible, include a comparison with automatically generated prompts (e.g., from a simple detector or a fixed grid) to assess performance in a more realistic setting.
- [FoundationShift Applied to Visual Language Foundation Model Classification of RCM (Supplementary Figure S7)] The UNI classification experiment is conducted on a single RCM mosaic, and the reported F1 scores depend on a cosine-similarity color mapping that the authors state was manually selected to maximize visibility of BCC nodules and minimize false negatives. This hand-tuning of the visualization threshold, combined with the single-image evaluation, means that the reported F1 values (ResNet-50: 0.46, UNI: 0.35, FoundationShift+UNI: 0.85) are not a reliable quantitative demonstration that FoundationShift 'significantly enhances' UNI. The claim should be supported by a pre-specified threshold, multiple test images, and ideally a statistical test across images. Without this, the UNI result remains anecdotal and should be presented as such rather than as a quantitative comparison.
minor comments (4)
- [FoundationShift Applied to Large Language Model Analysis of OCT] There is a typo in 'promt' (should be 'prompt') and the model name 'GPT-o4' appears in the text, which should be 'GPT-4o' to be consistent with the rest of the paper. Please also correct 'utilzing' and other small spelling errors throughout the manuscript.
- [Data availability] The paper states that all data needed to evaluate the conclusions are in the paper and/or Supplementary Materials, but the RCM data and annotation spreadsheets are provided via external Google Drive links. Please make the data availability statement more explicit by describing what files are included, and consider depositing the code and processed data in a persistent repository to improve reproducibility.
- [Theoretical Discussion (Supplementary Figure S2)] The KL divergence is computed on VGG16 embeddings, but the manuscript does not specify which layer or pooling is used, nor whether the values are symmetric or averaged across images. Providing these details would make the analysis reproducible and would clarify the interpretation of the reported divergence values.
- [OCT Tissue Segmentation and Statistical Analysis] The paper reports multiple paired Wilcoxon tests across three segmentation models but does not mention any multiple-testing correction. The p-values are extremely small, so the conclusions are unlikely to change, but a brief note about this would strengthen the statistical reporting.
Circularity Check
RCM cell-segmentation comparison is circular: the 4,503-cell ground truth was initialized by Hover-Net on virtual H&E, the exact pipeline being scored.
-
self definitional
[Methods, 'RCM Cell Segmentation and Statistical Analysis'; Figure 2c]
"First, we ran Hover-Net on the virtually stained dataset, then manually adjusted the results to create a ground truth consisting of 4,503 cells. We acknowledge that generating ground truth in this method may skew results in favor of Hover-Net."
The method being scored in Figure 2c is precisely domain transfer + Hover-Net on virtual H&E images. The reference standard used to score it was produced by running that same model on that same input modality and then manually adjusting the outputs. The reported DQ/SQ/PQ superiority over CellProfiler therefore compares CellProfiler against a ground truth initialized from Hover-Net's own segmentations; any systematic Hover-Net bias (oversegmentation, boundary placement, missed nuclei) can survive manual adjustment. The 50-cell CellProfiler-based control (Dice 0.9 between the two ground truths) covers about 1% of the 4,503 cells and measures mask overlap, not the DQ/SQ/PQ quantities used for the headline comparison, so it does not break the circularity.
full rationale
The OCT tissue-segmentation claim is evaluated against expert ground truth (two independent experts, inter-expert Dice 0.85), with the OCT2Hist domain-transfer model taken from prior published work. Although that prior work shares an author, the model was not fitted to the 95 test sections, and the with/without domain-transfer comparison is a genuine empirical evaluation; this is self-citation but not circularity. The PLIP, UNI, and ChatGPT experiments also do not involve any parameter fitted to the reported metrics. The one substantive circular step is the RCM cell-segmentation ground truth: Hover-Net's own outputs on virtual H&E initialize the 4,503-cell reference standard that is then used to show Hover-Net outperforms CellProfiler. The paper explicitly acknowledges this ('may skew results in favor of Hover-Net') and offers a 50-cell control, but the control is too small and does not directly bound the DQ/SQ/PQ differences used in the main comparison. Because RCM is one of two modalities supporting the central claim, this makes the RCM cell-segmentation result partially circular while leaving the OCT and language-model results with independent content. Overall score 6: partial circularity, not total.
Assumptions & free parameters
free parameters (3)
- Bounding box height factor =
1.15
- UNI cosine similarity threshold / color mapping =
not reported
- CellProfiler parameters =
not reported
assumptions (5)
- domain assumption KL divergence on VGG16 embeddings measures how well a domain-transferred image aligns with the CPath training domain, and this alignment predicts downstream model accuracy.
- ad hoc to paper The ground truth for RCM cell segmentation, created by running Hover-Net on virtual H&E and manually adjusting outputs, is unbiased for evaluating Hover-Net.
- domain assumption The OCT2Hist and Li et al. domain transfer models produce virtual H&E images that preserve the structures needed by downstream segmentation and classification models.
- domain assumption Expert OCT annotations with hair-follicle regions excluded are a reliable ground truth for epidermis segmentation.
- standard math Wilcoxon signed-rank test assumptions hold for paired Dice scores across 95 sections.
Cite this review
Pith. "Pith review of Leveraging Computational Pathology AI for Noninvasive Optical Imaging Analysis Without Retraining." pith.science (2026). https://pith.science/paper/2DZLKFGZ
@misc{pith2026241111613,
author = {Pith},
title = {Pith review of: Leveraging Computational Pathology AI for Noninvasive Optical Imaging Analysis Without Retraining},
year = {2026},
howpublished = {\url{https://pith.science/paper/2DZLKFGZ}},
note = {Machine review of arXiv:2411.11613}
}
read the original abstract
Noninvasive optical imaging modalities can probe patient's tissue in 3D and over time generate gigabytes of clinically relevant data per sample. There is a need for AI models to analyze this data and assist clinical workflow. The lack of expert labelers and the large dataset required (>100,000 images) for model training and tuning are the main hurdles in creating foundation models. In this paper we introduce FoundationShift, a method to apply any AI model from computational pathology without retraining. We show our method is more accurate than state of the art models (SAM, MedSAM, SAM-Med2D, CellProfiler, Hover-Net, PLIP, UNI and ChatGPT), with multiple imaging modalities (OCT and RCM). This is achieved without the need for model retraining or fine-tuning. Applying our method to noninvasive in vivo images could enable physicians to readily incorporate optical imaging modalities into their clinical practice, providing real time tissue analysis and improving patient care.
Figures
Reference graph
Works this paper leans on
-
[1]
Fisher, J., Siegel, D. M. & Markowitz, O. Clinical utility of bedside multibeam optical coherence tomography imaging in a patient with multiple basal cell carcinomas. Dermatol. Surg. 44, 874–876 (2018)
work page 2018
-
[2]
Ulrich, M. et al. The sensitivity and specificity of optical coherence tomography for the assisted diagnosis of nonpigmented basal cell carcinoma: an observational study. Br. J. Dermatol. 173, 428–435 (2015)
work page 2015
-
[3]
Li, G. et al. High-definition optical coherence tomography in the diagnosis of basal cell carcinoma evaluated by an experienced versus inexperienced investigator. J. Clin. & Exp. Dermatol. Res. 5, 1–4 (2014)
work page 2014
-
[4]
Kuck, M. et al. Evaluation of optical coherence tomography as a non-invasive diagnostic tool in cutaneous wound healing. Ski. Res. Technol. 20, 1–7 (2014)
work page 2014
-
[5]
Tsai, C.-Y .et al. Submicron spatial resolution optical coherence tomography for visualising the 3d structures of cells cultivated in complex culture systems. Sci. Reports 11, 3492 (2021)
work page 2021
-
[6]
Lin, C.-H. et al. Rapid measurement of epidermal thickness in oct images of skin. Sci. Reports 14, 2230 (2024)
work page 2024
-
[7]
Kumar, P., Dhara, S., Gope, A., Chatterjee, J. & Mandal, S. Deep learning based skin-layer segmentation for characterizing cutaneous wounds from optical coherence tomography images. In 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) , 1–4 (IEEE, 2023)
work page 2023
-
[8]
Shi, P. et al. Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation. Diagnostics 13, 1947 (2023)
work page 2023
Show all 40 references
-
[9]
Xu, H. et al. A whole-slide foundation model for digital pathology from real-world data. Nature 1–8 (2024)
2024
-
[10]
S., Churchill, S., Tan, A
Kohane, I. S., Churchill, S., Tan, A. L. M., Vella, M. & Perry, C. L. The digital–physical divide for pathology research. The Lancet Digit. Heal. 5, e859–e861 (2023)
2023
-
[11]
Huang, Z. et al. Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images. NPJ Precis. Oncol. 7, 14 (2023)
2023
-
[12]
Cumulative sum analysis for the learning curve of optical coherence tomography assisted diagnosis of basal cell carcinoma
Eva, V .et al. Cumulative sum analysis for the learning curve of optical coherence tomography assisted diagnosis of basal cell carcinoma. Acta dermato-venereologica 100 (2020)
2020
-
[13]
& Loy, C
Zhou, K., Liu, Z., Qiao, Y ., Xiang, T. & Loy, C. C. Domain generalization: A survey. IEEE Transactions on Pattern Analysis Mach. Intell. 45, 4396–4415 (2022)
2022
-
[14]
Noninvasive virtual biopsy using micro-registered optical coherence tomography (oct) in human subjects
Winetraub, Y .et al. Noninvasive virtual biopsy using micro-registered optical coherence tomography (oct) in human subjects. Sci. Adv. 10, eadi5794 (2024)
2024
-
[15]
Li, J. et al. Biopsy-free in vivo virtual histology of skin using deep learning. Light. Sci. & Appl. 10, 233 (2021)
2021
-
[16]
Cao, R. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat. biomedical engineering 7, 124–134 (2023)
2023
-
[17]
Liu, Z. et al. Virtual formalin-fixed and paraffin-embedded staining of fresh brain tissue via stimulated raman cyclegan model. Sci. Adv. 10, eadn3426 (2024). 7/14
2024
-
[18]
Y .et al
Park, W. Y .et al. Open-top bessel beam two-photon light sheet microscopy for three-dimensional pathology. Elife 12, RP92614 (2024)
2024
-
[19]
Warner, E. et al. Multimodal machine learning in image-based and clinical biomedicine: Survey and prospects. Int. J. Comput. Vis. 1–17 (2024)
2024
-
[20]
& Abtahi, S.-H
Nouri, H., Nasri, R. & Abtahi, S.-H. Addressing inter-device variations in optical coherence tomography angiography: will image-to-image translation systems help? Int. J. Retin. Vitreous 9, 51 (2023)
2023
-
[21]
& Jiang, C
Wang, J., Zong, Y ., He, Y ., Shi, G. & Jiang, C. Domain adaptation-based automated detection of retinal diseases from optical coherence tomography images. Curr. Eye Res. 48, 836–842 (2023)
2023
-
[22]
& Liu, M
Guan, H. & Liu, M. Domain adaptation for medical image analysis: a survey. IEEE Transactions on Biomed. Eng. 69, 1173–1185 (2021)
2021
-
[23]
Kirillov, A. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 4015–4026 (2023)
2023
-
[24]
Ma, J. et al. Segment anything in medical images. Nat. Commun. 15, 654 (2024)
2024
-
[25]
Cheng, J. et al. Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)
2023 arXiv
-
[26]
Graham, S. et al. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. image analysis 58, 101563 (2019)
2019
-
[27]
Carpenter, A. E. et al. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome biology 7, 1–11 (2006)
2006
-
[28]
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J. & Zou, J. A visual–language foundation model for pathology image analysis using medical twitter. Nat. medicine 29, 2307–2316 (2023)
2023
-
[29]
Achiam, J. et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
2023 arXiv
-
[30]
Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Medicine 30, 850–862 (2024)
2024
-
[31]
& Wong, T
Kang, L., Li, X., Zhang, Y . & Wong, T. T. Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining. Photoacoustics 25, 100308 (2022)
2022
-
[32]
Shi, L. et al. Thickv-stain: Unprocessed thick tissues virtual staining for rapid intraoperative histology. In Medical Imaging with Deep Learning (2024)
2024
-
[33]
Song, A. H. et al. Analysis of 3d pathology samples using weakly supervised ai. Cell 187, 2502–2520 (2024)
2024
-
[34]
Del Amor, R. et al. Automatic segmentation of epidermis and hair follicles in optical coherence tomography images of normal skin by convolutional neural networks. Front. Medicine 7, 220 (2020)
2020
-
[35]
Cheng, S. et al. Enhanced multiscale human brain imaging by semi-supervised digital staining and serial sectioning optical coherence tomography. Res. Sq. (2024)
2024
-
[36]
Park, E. S. Skin-layer analysis using optical coherence tomography (oct). Med. Lasers 3, 1–4 (2014)
2014
-
[37]
Taghavikhalilbad, A. et al. Semi-automated localization of dermal epidermal junction in optical coherence tomography images of skin. Appl. optics 56, 3116–3121 (2017)
2017
-
[38]
Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Transactions on Med. Imaging 36, 1550–1560, DOI: 10.1109/TMI.2017.2677499 (2017)
2017
-
[39]
Jin, C.-B. et al. Deep ct to mr synthesis using paired and unpaired data. Sensors 19, 2361 (2019)
2019
-
[40]
Describe the Image
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 770–778 (2016). 8/14 Domain Transfer Model for OCT: OCT2Hist Figure S1. OCT2Hist: domain transfer from OCT to v...
2016
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.