REVIEW 3 major objections 4 minor 31 references
Segment Anything for Cell Tracking
T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A frozen, unmodified SAM2 model can link cell masks across time-lapse microscopy frames and detect mitotic divisions without any dataset-specific training, performing on par with supervised trackers across the Cell Tracking Challenge.
desk verdict A genuinely zero-shot 2D cell-linking method built on SAM2, wrapped in a 3D pipeline that fine-tunes on the target data and reports training-set scores; the 2D part deserves a referee, the 3D claims need revisions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is SAM2's promptable video mask propagation: a short sequence of two image patches, a mask-derived bounding box, and positive/negative point prompts let the frozen model hallucinate the cell mask in the previous frame, while the model's memory encoder produces feature vectors whose cosine similarity links candidate cells in large-scale tracking. For 3D volumes, SAM-Med3D—a medical foundation model not trained on microscopy—is fine-tuned on rough watershed masks so a single click generates a 3D segmentation mask.
What would settle it
Run the authors' released code on the official CTC test set for Fluo-N3DL-TRIC and Fluo-N3DL-TRIF and compare the reported training-set SEG/TRA values against the test-set values; if the test-set scores drop materially, the central 3D competitive claim is not supported. A second check: apply the zero-shot linking pipeline to a held-out microscopy dataset with ground-truth lineages and see whether LNK remains top-tier without any threshold tuning.
Extended reading notes
Core claim
The paper's central claim is that a fixed, pre-trained SAM2 model—used without any fine-tuning or dataset-specific adaptation—can serve as the core of a cell-tracking pipeline. Given a known mask at time t, the model receives a crop of frames t and t−1 plus bounding-box and point prompts, and predicts the corresponding mask at t−1; comparing this prediction to pre-segmented masks links the cell across frames, and two cells linking back to the same parent signals mitosis. The authors report top-3 average LNK score over 13 blind test datasets, and for large-scale 3D data they add SAM-Med3D, fine-tuned on rough watershed-generated masks, to segment cells during forward tracking, achieving secon
Load-bearing premise
For the large-scale 3D results, the paper assumes that scores measured on the CTC training set after fine-tuning SAM-Med3D on rough masks from those same sequences are representative of what the method would achieve on the official blind test set, an equivalence asserted without tabulated test-set scores.
Editorial extensions
If this is right
- Cell tracking on new microscopy datasets no longer requires collecting manual segmentation and tracking annotations; a frozen foundation model plus hand-set thresholds may suffice.
- Tracking quality becomes directly tied to the quality of the initial segmentation masks, since SAM2 propagates prompts rather than learning dataset-specific appearance priors.
- Because the pipeline is prompt-based, users can interactively correct detections in one frame and have corrections propagate backward or forward, enabling human-in-the-loop refinement.
- Large-scale 3D time-lapse data can be processed patch-wise in parallel, making tracking of thousands of cells computationally feasible on a single GPU.
- If the method holds on official test sets, it offers a strong baseline for evaluating new supervised trackers, since it uses no training data of any kind.
Reading between the lines
- If confirmed on blind tests, this zero-shot approach could substantially lower the entry barrier for cell tracking in understudied organisms or unusual imaging modalities where labeled data are scarce.
- The same prompt-based linking mechanism could transfer to non-microscopy object tracking tasks that supply mask sequences, since SAM2's video capabilities are not microscopy-specific.
- The reliance on pre-segmented masks means the pipeline inherits their errors; combining the linking step with self-supervised segmentation refinement could remove the need for even the rough watershed masks in the 3D setting.
- The method's weakness on large appearance changes and ID switches suggests that integrating whole-structure motion prediction, as the authors state they plan to do, might close the gap to global optimization approaches on BIO scores.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a cell tracking pipeline built on the pre-trained SAM2 foundation model. For 2D and small 3D data, the method links pre-segmented masks by propagating a mask backward in time through SAM2 prompted with the previous frame's mask, recovering missed detections and detecting mitoses when two daughter masks link to one parent. For large-scale 3D+t data, the pipeline combines SAM2-based similarity linking with SAM-Med3D, which is fine-tuned on rough watershed masks generated from the target sequences, to produce volumetric segmentation and tracking. The authors report top-3 LNK ranks on several CTC blind test datasets and competitive SEG/TRA scores on two large 3D datasets, and claim the framework is zero-shot, fully unsupervised, and requires no fine-tuning or dataset-specific adaptation.
Significance. The 2D linking contribution is potentially valuable: it demonstrates that an unmodified pre-trained video segmentation model can be prompted to link cells across diverse microscopy modalities without any labeled training, and the evaluation on blind test datasets gives credible evidence. The public code is a concrete strength. However, the advertised scope is significantly broader than what is actually demonstrated. The large-3D pipeline fine-tunes SAM-Med3D on masks derived from the same sequences that are later evaluated, and the reported 3D scores are computed on the CTC training set rather than the official test set. These two issues undercut the central claims of 'zero-shot,' 'without fine-tuning,' and 'competitive accuracy in large-scale 3D' as stated in the abstract. The paper is therefore a solid candidate after substantial revision that aligns claims with evidence and supplies missing test-set measurements.
major comments (3)
- [Abstract; §2.2; §3.3] The central claim of zero-shot / no-fine-tuning operation is internally contradicted. Section 2.2 states 'We then fine-tune SAM-Med3D using these rough masks' where the rough masks are generated from the target sequences, while Section 3.3 claims 'we apply it directly to the training set without parameter fine-tuning'. Fine-tuning SAM-Med3D on target-data-derived masks is dataset-specific adaptation even if the masks are unannotated. This directly contradicts the abstract's 'without fine-tuning' and 'eliminating the need for dataset-specific adaptation.' Please either remove the fine-tuning step, report results without it, or revise the abstract/contributions to claim only 'no manual annotations' for the 2D path and clearly label the large-3D variant as using unsupervised fine-tuning on target data.
- [Table 2; §3.3] The large-3D comparison is not valid as presented. Table 2 reports SEG/TRA scores on the CTC training set, while the competing methods are evaluated on the official CTC test set. The statement that 'training and test set scores are highly similar in our case' is not supported by tabulated test-set numbers; supplementary videos are qualitative and cannot substitute for quantitative evaluation. Please provide actual test-set SEG/TRA scores under the official CTC protocol, or explicitly relabel Table 2 as a non-comparable training-set benchmark. As written, the 'third in SEG / second in TRA' claim is not established.
- [§3.2; Table 1] The 2D linking claim is 'top 3 in LNK score on average across all 13 datasets,' but Table 1 shows only six of the 13 datasets and reports ranks rather than the average LNK value. It is unclear whether 'top 3' refers to mean LNK score or mean rank. Please provide the full per-dataset table (or a supplementary table) with LNK/BIO values and state the averaging procedure so the 2D claim is verifiable. Without this, the headline 2D result is not fully reproducible from the manuscript.
minor comments (4)
- [§2.2] The search radius τ and patch side length d are introduced as predefined but no values are given in the experiments. Please report the exact values used for each dataset.
- [§2.2] The mitosis threshold (similarity difference below 0.1) and the linking threshold (0.8) are stated as fixed, but the paper also says the latter 'might need adjustments in other circumstances.' Please clarify whether these thresholds were fixed across all datasets or tuned per dataset, and how they relate to the claim of no dataset-specific adaptation.
- [Table 1/Table 2] In Table 1, the superscript numbers are method identifiers, but in Table 2 the bold row for the proposed method is not explicitly labeled with a superscript. Please add a note explaining the notation consistently.
- [General] The supplementary videos are referenced but not accessible in the manuscript; please ensure the supplementary material is available to reviewers and readers, and consider adding quantitative per-sequence results in the supplementary text.
Circularity Check
Large-scale 3D results are produced by fine-tuning SAM-Med3D on rough masks from the same sequences that are later scored, contradicting the zero-shot/no-fine-tuning claim.
-
fitted input called prediction
[Section 2.2 (Large-scale 3D) and Section 3.3 / Table 2]
"We then fine-tune SAM-Med3D using these rough masks, enabling it to generate 3D segmentation masks with a single click on the cell area. ... Since our approach is fully-unsupervised, meaning no ground truth information from the training set is used, we apply it directly to the training set without parameter fine-tuning and evaluate its performance using ground truth annotations."
The fine-tuning step adapts SAM-Med3D to rough masks extracted from the exact sequences (Fluo-N3DL-TRIC/TRIF) whose ground-truth SEG/TRA are then reported in Table 2. The paper calls this 'without parameter fine-tuning' and 'zero-shot,' but the 3D segmentation weights are fitted to the evaluated data before scoring. The reported 3D scores are therefore training-set measurements after target-data adaptation, not zero-shot predictions; the subsequent assertion that 'training and test set scores are highly similar' is unsupported by any tabulated test-set numbers, so the competitive-accuracy claim rests on this fitted evaluation. The 'prediction' is statistically dependent on the input data it purports to generalize from.
full rationale
The 2D linking and small-3D results are genuinely independent: SAM2 is used as-is with hand-set prompts and no training on target data. The core circularity is confined to the large-scale 3D contribution, where the model is fine-tuned on rough masks from the same sequences scored in Table 2, and then the paper asserts without tabulated evidence that this transfers to test data. Since this is the basis for the abstract's 'large-scale 3D without dataset-specific adaptation' claim, the central claim partially reduces to a fit on evaluated data. No self-citation chain or uniqueness-theorem circularity is present.
Assumptions & free parameters
free parameters (4)
- similarity threshold =
0.8
- mitosis difference threshold =
0.1
- search radius tau =
not specified
- patch side length d =
not specified
assumptions (4)
- domain assumption SAM2's video mask propagation transfers to microscopy image patches despite being trained on natural video.
- domain assumption Cosine similarity of SAM2 memory-encoded features is a reliable proxy for cell identity across frames.
- domain assumption SAM-Med3D, trained on medical images, can be fine-tuned on rough watershed masks to produce accurate 3D cell masks.
- domain assumption CTC ground-truth lineage annotations are correct and the AOGM/TRA/SEG metrics are appropriate for comparing tracking methods.
Cite this review
Pith. "Pith review of Segment Anything for Cell Tracking." pith.science (2026). https://pith.science/paper/KPFFUB2E
@misc{pith2026250909943,
author = {Pith},
title = {Pith review of: Segment Anything for Cell Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/KPFFUB2E}},
note = {Machine review of arXiv:2509.09943}
}
read the original abstract
Tracking cells and detecting mitotic events in time-lapse microscopy image sequences is a crucial task in biomedical research. However, it remains highly challenging due to dividing objects, low signal-tonoise ratios, indistinct boundaries, dense clusters, and the visually similar appearance of individual cells. Existing deep learning-based methods rely on manually labeled datasets for training, which is both costly and time-consuming. Moreover, their generalizability to unseen datasets remains limited due to the vast diversity of microscopy data. To overcome these limitations, we propose a zero-shot cell tracking framework by integrating Segment Anything 2 (SAM2), a large foundation model designed for general image and video segmentation, into the tracking pipeline. As a fully-unsupervised approach, our method does not depend on or inherit biases from any specific training dataset, allowing it to generalize across diverse microscopy datasets without finetuning. Our approach achieves competitive accuracy in both 2D and large-scale 3D time-lapse microscopy videos while eliminating the need for dataset-specific adaptation.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Nature Methods (2025)
Archit, A., Freckmann, L., Nair, S., Khalid, N., Hilt, P., Rajashekar, V., Freitag, M., Teuber, C., Buckley, G., von Haaren, S., et al.: Segment anything for mi- croscopy. Nature Methods (2025)
2025
-
[2]
In: 17th European Conference on Computer Vision, ECCV
Ben-Haim, T., Raviv, T.R.: Graph neural network for cell tracking in microscopy videos. In: 17th European Conference on Computer Vision, ECCV. pp. 610–626. Springer (2022)
2022
-
[3]
bioRxiv:2024.09.02.610652v1 (2024)
Bragantini, J., Theodoro, I., Zhao, X., Huijben, T.A., Hirata-Miyasaki, E., VijayKumar, S., Balasubramanian, A., Lao, T., Agrawal, R., Xiao, S., et al.: Ultrack: pushing the limits of cell tracking across biological scales. bioRxiv:2024.09.02.610652v1 (2024)
2024
-
[4]
Nature Communications11(1), 6254 (2020)
Cao, J., Guan, G., Ho, V.W.S., Wong, M.K., Chan, L.Y., Tang, C., Zhao, Z., Yan, H.: Establishment of a morphological atlas of the caenorhabditis elegans embryo using deep-learning-based 4d segmentation. Nature Communications11(1), 6254 (2020)
2020
-
[5]
Cell147(5), 992–1009 (2011)
Friedl, P., Alexander, S.: Cancer invasion and the microenvironment: plasticity and reciprocity. Cell147(5), 992–1009 (2011)
2011
-
[6]
Nature reviews Molecular cell biology10(7), 445–457 (2009)
Friedl, P., Gilmour, D.: Collective cell migration in morphogenesis, regeneration and cancer. Nature reviews Molecular cell biology10(7), 445–457 (2009)
2009
-
[7]
In: 18th European Conference on Computer Vision, ECCV
Gallusser, B., Weigert, M.: Trackastra: Transformer-based cell tracking for live- cell microscopy. In: 18th European Conference on Computer Vision, ECCV. pp. 467–484. Springer (2024)
2024
-
[8]
In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part I 22
Hayashida, J., Bise, R.: Cell tracking with deep learning for cell detection and motion estimation in low-frame-rate. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part I 22. pp. 397–405. Springer (2019)
2019
Show all 31 references
-
[9]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Hayashida, J., Nishimura, K., Bise, R.: Mpm: Joint representation of motion and position map for cell tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3823–3832 (2020)
2020
-
[10]
bioRxiv:2023.11.17.567630v5 (2024)
Israel, U., Marks, M., Dilip, R., Li, Q., Yu, C., Laubscher, E., Li, S., Schwartz, M., Pradhan, E., Ates, A., et al.: A foundation model for cell segmentation. bioRxiv:2023.11.17.567630v5 (2024)
2023
-
[11]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)
2023
-
[12]
arXiv preprint arXiv:2212.03465 (2022)
Lee, G., Kim, S., Kim, J., Yun, S.: Mediar: Harmony of data-centric and model- centric for multi-modality microscopy.(2022). arXiv preprint arXiv:2212.03465 (2022)
2022 arXiv
-
[13]
IEEE Access10, 77147–77157 (2022)
Löffler, K., Mikut, R.: Embedtrack—simultaneous cell segmentation and tracking through learning offsets and clustering bandwidths. IEEE Access10, 77147–77157 (2022)
2022
-
[14]
In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI)
Lux, F., Matula, P.: Cell tracking based on integer linear programming and prob- ability scores. In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2024)
2024
-
[15]
Ma, J., Kim, S., Li, F., Baharoon, M., Asakereh, R., Lyu, H., Wang, B.: Segment anythinginmedicalimagesandvideos:Benchmarkanddeployment.arXivpreprint arXiv:2408.03322 (2024) Segment Anything for Cell Tracking 11
2024 arXiv
-
[16]
Magnusson, K.E.: Segmentation and tracking of cells and particles in time-lapse microscopy. Ph.D. thesis, KTH Royal Institute of Technology (2016)
2016
-
[17]
IEEE Transactions on Medical Imaging34(4), 911–929 (2014)
Magnusson, K.E., Jaldén, J., Gilbert, P.M., Blau, H.M.: Global linking of cell tracks using the viterbi algorithm. IEEE Transactions on Medical Imaging34(4), 911–929 (2014)
2014
-
[18]
Nature biotech- nology41(1), 44–49 (2023)
Malin-Mayor, C., Hirsch, P., Guignard, L., McDole, K., Wan, Y., Lemon, W.C., Kainmueller, D., Keller, P.J., Preibisch, S., Funke, J.: Automated reconstruction of whole-embryo cell lineages by learning from sparse annotations. Nature biotech- nology41(1), 44–49 (2023)
2023
-
[19]
Nature Methods 20(7), 1010–1020 (2023)
Maška, M., Ulman, V., Delgado-Rodriguez, P., Gómez-de Mariscal, E., Nečasová, T., Guerrero Peña, F.A., Ren, T.I., Meyerowitz, E.M., Scherr, T., Löffler, K., et al.: The cell tracking challenge: 10 years of objective benchmarking. Nature Methods 20(7), 1010–1020 (2023)
2023
-
[20]
PloS one10(12), e0144959 (2015)
Matula, P., Maška, M., Sorokin, D.V., Matula, P., Ortiz-de Solórzano, C., Kozubek, M.: Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PloS one10(12), e0144959 (2015)
2015
-
[21]
Biology Open7(7), bio027730 (2018)
May, M., Denecke, B., Schroeder, T., Götz, M., Faissner, A.: Cell tracking in vitro reveals that the extracellular matrix glycoprotein tenascin-c modulates cell cycle length and differentiation in neural stem/progenitor cells of the developing mouse spinal cord. Biology Open7(...
2018
-
[22]
https://doi.org/10.5281/zenodo.14914189
Miles, A., jakirkham, Hamman, J., Orfanos, D.P., Stansby, D., Bussonnier, M., Moore, J., Bennett, D., Augspurger, T., Rzepka, N., et al.: zarr-developers/zarr- python: v3.0.4 (2025). https://doi.org/10.5281/zenodo.14914189
2025 doi
-
[23]
bioRxiv:803205v2 (2019)
Moen, E., Borba, E., Miller, G., Schwartz, M., Bannon, D., Koe, N., Camplisson, I., Kyme, D., Pavelchek, C., Price, T., et al.: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning. bioRxiv:803205v2 (2019)
2019
-
[24]
arXiv preprint arXiv:2401.13220 (2024)
Na, S., Guo, Y., Jiang, F., Ma, H., Huang, J.: Segment any cell: A sam-based auto-prompting fine-tuning framework for nuclei segmentation. arXiv preprint arXiv:2401.13220 (2024)
2024 arXiv
-
[25]
arXiv preprint arXiv:2408.00714 (2024)
Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)
2024 arXiv
-
[26]
PloS one9(2), e90036 (2014)
Stegmaier, J., Otte, J.C., Kobitski, A., Bartschat, A., Garcia, A., Nienhaus, G.U., Strähle, U., Mikut, R.: Fast segmentation of stained nuclei in terabyte-scale, time resolved 3d microscopy image stacks. PloS one9(2), e90036 (2014)
2014
-
[27]
Nature Methods (2025)
Stringer, C., Pachitariu, M.: Cellpose3: one-click image restoration for improved cellular segmentation. Nature Methods (2025)
2025
-
[28]
Nature Methods14(12), 1141–1152 (2017)
Ulman, V., Maška, M., Magnusson, K.E., Ronneberger, O., Haubold, C., Harder, N., Matula, P., Matula, P., Svoboda, D., Radojevic, M., et al.: An objective com- parison of cell-tracking algorithms. Nature Methods14(12), 1141–1152 (2017)
2017
-
[29]
arXiv preprint arXiv:2310.15161 (2023)
Wang, H., Guo, S., Ye, J., Deng, Z., Cheng, J., Li, T., Chen, J., Su, Y., Huang, Z., Shen, Y., et al.: Sam-med3d: towards general-purpose segmentation models for volumetric medical images. arXiv preprint arXiv:2310.15161 (2023)
2023 arXiv
-
[30]
In: 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC)
Weigert, M., Schmidt, U.: Nuclei instance segmentation and classification in histopathology images with stardist. In: 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC). pp. 1–4. IEEE (2022)
2022
-
[31]
CoRR (2024)
Zhu, J., Qi, Y., Wu, J.: Medical sam 2: Segment medical images as video via segment anything model 2. CoRR (2024)
2024
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.