REVIEW 2 major objections 5 minor 35 references
Mammography foundation-model transfer is not explained by mammography exposure alone; dataset-level out-of-distribution evaluation is the decisive test.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 12:20 UTC pith:IX43H6FT
load-bearing objection Solid multi-site mammography FM benchmark: VLMs lead mean OOD, mammography exposure alone does not, DINOv3 stays hard to beat, and per-dataset heterogeneity is the real takeaway. the 2 major comments →
Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Under a single frozen-backbone linear-probe protocol spanning density, BI-RADS, and cancer status, mammography-specific vision-language models—especially Mammo-FM and MaMA—achieve the highest mean out-of-distribution AUROC, but that advantage is not explained by mammography exposure alone: natural-image DINOv3 stays competitive, single-source mammography adaptation often fails to help, and even the best models show large, task-dependent swings across individual external datasets. Useful representations can preserve clinical signal while retaining dataset and acquisition structure, so dataset-level OOD evaluation—not source performance or pretraining domain alone—is the central criterion for
What carries the argument
A unified frozen-backbone linear-probe benchmark: fifteen foundation encoders are frozen; a task-specific linear head is trained only on one of three source datasets (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer); evaluation uses macro or binary AUROC on twelve task-compatible external datasets after label harmonization, with exam-level max pooling for BI-RADS and cancer. This isolates representation quality under domain shift without confounding full fine-tuning.
Load-bearing premise
The study treats frozen linear-probe AUROC after cross-dataset label harmonization as a fair proxy for representation quality under real domain shift, even though cancer labels mix different clinical definitions and the protocol does not measure the best performance full fine-tuning could reach.
What would settle it
If full fine-tuning on the same sources reversed the model ranking and made mammography-adapted vision-only models match or beat the mammography vision-language models with uniform gains across external datasets—including film and contrast-enhanced sets—then the claim that pretraining type and dataset-level OOD evaluation, rather than mammography exposure or source scores, are the decisive criteria would fail.
If this is right
- Mean OOD AUROC is an incomplete report card; model selection for deployment should report per-dataset scores.
- Choosing a backbone solely because it was pretrained on mammography can mislead; language alignment and pretraining diversity matter more than modality exposure alone.
- Strong natural-image self-supervised encoders remain necessary baselines in mammography representation work.
- Feature-space domain structure can coexist with clinical utility; full domain invariance is not required for useful transfer.
- Claims about mammography foundation models should include multi-country, multi-vendor OOD panels rather than single held-out splits.
Where Pith is reading between the lines
- Linear probes may understate how much adaptation could close dataset-specific gaps; full fine-tuning stratified by acquisition type (film, contrast-enhanced, tomosynthesis) would test whether rankings stabilize.
- Heterogeneous cancer-status definitions across datasets may make some OOD swings partly definitional; standardizing binary cancer proxies before ranking would isolate true representation transfer.
- If metadata-derived text (density, view) drives MaMA’s density edge while clinical-report text drives Mammo-FM’s BI-RADS and cancer edge, pretraining text design becomes a controllable lever for task-targeted transfer.
- Multi-site prospective screening cohorts would be the natural next stress test beyond public archives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper benchmarks 15 foundation-model backbones as frozen mammography feature extractors under a unified linear-probe protocol. Using three source datasets (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer) and up to 12 task-compatible OOD datasets after label harmonization, it compares natural-image SSL, general radiology SSL, mammography-adapted SSL, general medical VLMs, and mammography-specific VLMs on density, BI-RADS, and cancer status. The central claims are that mammography-specific VLMs (especially Mammo-FM and MaMA) achieve the strongest mean OOD AUROC, that mammography exposure alone does not explain robustness (DINOv3 remains competitive; single-source adaptation can hurt), and that even leading models show heterogeneous per-dataset OOD performance, so dataset-level OOD evaluation should be a primary criterion. Supporting analyses include ID–OOD scatter, family averages, per-dataset heatmaps, and UMAP feature inspection.
Significance. If the comparative ranking holds under the stated protocol, the paper provides a timely and practically useful reference for mammography representation learning. The multi-task, multi-country design, explicit leakage control (RSNA-site2 only), public code, and honest limitation that linear probes measure representation quality rather than best absolute performance are real strengths. The finding that mammography-specific VLMs lead while DINOv3 remains hard to beat, and that continued pretraining on a single mammography corpus does not reliably improve OOD transfer, is actionable for model selection and for future pretraining design. Dataset-level heterogeneity and feature-space structure further push the field beyond single-number leaderboards.
major comments (2)
- §3.1 and Table 1: cancer-status OOD averages mix heterogeneous label definitions (BI-RADS-derived positives in VinDr, screening vs cancer-enriched cohorts, coarse normal/abnormal labels). The paper acknowledges this and reports a weak ID–OOD rank correlation (ρ=0.61) and a slightly positive mean gap, but still presents mean cancer OOD AUROC as a primary ranking axis in Table 3 and the abstract. Please either (i) restrict the cancer mean to a clearly defined subset of comparable targets, or (ii) demote cancer mean OOD to a secondary/exploratory metric and lead with density and BI-RADS, so the headline ranking is not driven by an averaged mixture of different tasks.
- §2.2 and §3.3: several mammography-specific models (MaMA, GLAM, VersaMammo, Mammo-CLIP, Mammo-FM) report pretraining exposure to EMBED, VinDr-Mammo, and/or RSNA—the same sources used as ID training sets. The protocol freezes the backbone and trains only a linear head, which limits direct leakage, but partial source overlap still weakens the claim that mammography exposure alone does not explain robustness and that OOD gains are purely from objective/alignment. Please add an explicit overlap table (model × source datasets) and, where possible, a leave-source-out or non-overlapping OOD subset analysis so the family comparison is not confounded by shared pretraining data.
minor comments (5)
- Table 1: the † exclusion rule for incomplete BI-RADS ordinal targets is stated in the caption but not fully reflected in the main text of §2.1; a short sentence listing which datasets are dropped from the final BI-RADS OOD mean would help.
- Figure 2: the zoom insets are useful but the error bars are hard to read at the printed scale; consider larger markers or a supplementary table of exact ID/OOD values with CIs.
- §2.3: exam-level max pooling of image probabilities is a reasonable clinical aggregation, but a brief sensitivity check (mean pooling or majority) would strengthen the claim that rankings are not aggregation-dependent.
- §3.5 / Figure 4: UMAP is qualitative; if space allows, a simple linear probe or silhouette score for density/BI-RADS vs manufacturer/view would make the “clinical signal retained while domain structure remains” claim more quantitative.
- Typos / consistency: “Ray DINO” vs “RayDINO” in Table 3; “Mammo-FMandMaMA” spacing in the abstract; arXiv IDs and access dates for challenge datasets should be standardized in the references.
Circularity Check
No circular derivation: comparative frozen linear-probe OOD benchmark on external public datasets, not a first-principles result forced by its inputs.
full rationale
This paper is an empirical multi-dataset benchmark, not a derivation that claims to predict or prove a quantity from first principles. Fifteen externally pretrained backbones are frozen; task heads are fit only on designated source splits (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer) and scored with AUROC on held-out ID and task-compatible OOD sets after label harmonization (Tables 1–3, Figures 2–3). Mean OOD rankings, family averages, and the claim that mammography exposure alone does not explain robustness are measurements under that protocol, not quantities recovered by construction from fitted constants. Pretraining-source overlap (e.g., MaMA/GLAM/Mammo-FM/VersaMammo seeing EMBED or VinDr during pretraining) and cancer-label heterogeneity are methodological confounds the paper partly acknowledges (§2.2, §3.1, §4); they do not make the reported OOD AUROCs tautological with the evaluation targets. Self-citations (e.g., Glocker et al. on chest FM bias) are background, not load-bearing uniqueness or ansatz imports. Feature UMAPs are qualitative inspection, not circular proofs. Per the stated criteria, there is no self-definitional step, no fitted input renamed as prediction, and no self-citation chain that forces the central comparative claims. Score 0 is appropriate.
Axiom & Free-Parameter Ledger
free parameters (4)
- image resize to 1024×768
- exam-level max-pool aggregation of image probabilities
- source-dataset assignment per task
- 1000 bootstrap resamples for 95% CIs
axioms (5)
- domain assumption Frozen-backbone linear probes measure transferable representation quality under domain shift.
- domain assumption Cross-dataset label harmonization yields comparable density, BI-RADS, and cancer tasks.
- domain assumption Macro one-vs-rest AUROC (density/BI-RADS) and binary AUROC (cancer) are appropriate primary metrics for ranking models under class imbalance and ordinal labels.
- domain assumption Public pretrained checkpoints can be compared as fixed feature extractors without re-pretraining under matched compute/data budgets.
- standard math Standard bootstrap confidence intervals and mean OOD averages over task-compatible datasets with finite AUROC are valid summaries.
read the original abstract
Foundation models are increasingly used as image feature extractors for mammography, but their robustness under external domain shift remains unclear. We benchmark 15 foundation-model backbones across breast density, BI-RADS severity, and cancer status using a unified frozen-backbone linear-probe protocol, training on 3 source datasets and evaluating on 12 task-compatible out-of-distribution (OOD) datasets after label harmonization. Mammography-specific vision-language models (Mammo-FM and MaMA) provide the strongest mean OOD performance, but robustness is not explained by mammography exposure alone. DINOv3 remains a competitive vision-only baseline, and mammography-adapted pretraining does not consistently improve generalization. Dataset-level analysis further shows that even leading models show heterogeneous performance across datasets. Feature-space inspection reveals that useful representations can preserve clinical signal while retaining dataset and acquisition structure. These findings highlight dataset-level OOD evaluation as a central criterion for assessing mammography representations. Our code is publicly available: https://github.com/biomedia-mira/mammo-ood.
Figures
Reference graph
Works this paper leans on
-
[1]
Data6(11), 111 (2021)
Alsolami, A.S., Shalash, W., Alsaggaf, W., et al.: King Abdulaziz University Breast Cancer Mammogram Dataset (KAU-BCMD). Data6(11), 111 (2021)
2021
-
[2]
American College of Radiology: ACR BI-RADS Atlas: Breast Imaging Reporting and Data System. 5th edn. American College of Radiology, Reston (2013)
2013
-
[3]
Data in Brief55, 110633 (2024)
Aqdar, K.B., Mustafa, R.K., Abdulqadir, Z.H., et al.: Mammogram mastery: A robust dataset for breast cancer detection and medical education. Data in Brief55, 110633 (2024)
2024
-
[4]
arXiv preprint arXiv:2108.07258 (2021)
Bommasani, R., Hudson, D.A., Adeli, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
Pith/arXiv arXiv 2021
-
[5]
JAMA Network Open4(8), e2119100 (2021)
Buda, M., Saha, A., Walsh, R., et al.: A data set and deep learning algorithm for the detection of masses and architectural distortions in digital breast tomosynthesis images. JAMA Network Open4(8), e2119100 (2021)
2021
-
[6]
Scientific Data10(1), 123 (2023)
Cai, H., Wang, J., Dan, T., et al.: An online mammography database with biopsy confirmed types. Scientific Data10(1), 123 (2023)
2023
-
[7]
arXiv preprint arXiv:2409.18119 (2024) 9
Du, Y., Onofrey, J., Dvornek, N.C.: Multi-view and multi-scale alignment for contrastive language-image pre-training in mammography. arXiv preprint arXiv:2409.18119 (2024) 9
Pith/arXiv arXiv 2024
-
[8]
arXiv preprint arXiv:2509.10344 (2025)
Du, Y., Chen, L., Dvornek, N.C.: GLAM: Geometry-guided local alignment for multi-view VLP in mammography. arXiv preprint arXiv:2509.10344 (2025)
Pith/arXiv arXiv 2025
-
[9]
arXiv preprint arXiv:2505.10579 (2025)
Germani, E., Türk, I.S., Zeineddine, F., Mourad, C., Albarqouni, S.: Bias and generalizability of foundation models across datasets in breast mammography. arXiv preprint arXiv:2505.10579 (2025)
Pith/arXiv arXiv 2025
-
[10]
In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, pp
Ghosh, S., Poynton, C.B., Visweswaran, S., Batmanghelich, K.: Mammo-CLIP: A vision language foundation model to enhance data efficiency and robustness in mammography. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, pp. 632–642. Springer Nature Switzerland, Cham (2024)
2024
-
[11]
arXiv preprint arXiv:2512.00198 (2025)
Ghosh, S., Joshi, V.P., Syed, R., et al.: Mammo-FM: Breast-specific foundational model for integrated mammographic diagnosis, prognosis, and reporting. arXiv preprint arXiv:2512.00198 (2025)
Pith/arXiv arXiv 2025
-
[12]
Radiology: Artificial Intelligence5(6), e230060 (2023)
Glocker, B., Jones, C., Roschewitz, M., Winzeck, S.: Riskofbiasinchestradiography deep learning foundation models. Radiology: Artificial Intelligence5(6), e230060 (2023)
2023
-
[13]
Radiology: Artificial Intelligence3(1), e200103 (2021)
Halling-Brown, M.D., Warren, L.M., Ward, D., et al.: OPTIMAM Mammography Image Database: A large-scale resource of mammography images and clinical data. Radiology: Artificial Intelligence3(1), e200103 (2021)
2021
-
[14]
In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp
He, K., Chen, X., Xie, S., et al.: Masked autoencoders are scalable vision learners. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979–15988. IEEE (2022)
2022
-
[15]
arXiv preprint arXiv:2509.20271 (2025)
Huang, F., Zhu, J., Yu, Y., et al.: A versatile foundation model for AI-enabled mammogram interpretation. arXiv preprint arXiv:2509.20271 (2025)
arXiv 2025
-
[16]
Radiology: Artificial Intelligence5(1), e220047 (2023)
Jeong, J.J., Vey, B.L., Bhimireddy, A., et al.: The EMory BrEast imaging Dataset (EMBED):Araciallydiverse, granulardatasetof3.4millionscreeninganddiagnostic mammographic images. Radiology: Artificial Intelligence5(1), e220047 (2023)
2023
-
[17]
Scientific Data12(1), 1479 (2025)
Kendall, E., Hajishafiezahramini, P., Hamilton, M., et al.: Full field digital mam- mography dataset from a population screening program. Scientific Data12(1), 1479 (2025)
2025
-
[18]
Scientific Data 9(1), 122 (2022)
Khaled, R., Helal, M., Alfarghaly, O., et al.: Categorized contrast enhanced mam- mography dataset for diagnostic and artificial intelligence research. Scientific Data 9(1), 122 (2022)
2022
-
[19]
arXiv preprint arXiv:2412.10372 (2024)
Khattak, M.U., Kunhimon, S., Naseer, M., Khan, S., Khan, F.S.: UniMed-CLIP: To- wards a unified image-text pretraining paradigm for diverse medical imaging modal- ities. arXiv preprint arXiv:2412.10372 (2024)
Pith/arXiv arXiv 2024
-
[20]
Scientific Data 4(1), 170177 (2017) 10
Lee, R.S., Gimenez, F., Hoogi, A., Miyake, K.K., et al.: A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific Data 4(1), 170177 (2017) 10
2017
-
[21]
European Radiology Experimental5(1), 40 (2021)
Loizidou, K., Skouroumouni, G., Pitris, C., Nikolaou, C.: Digital subtraction of temporally sequential mammograms for improved detection and classification of mi- crocalcifications. European Radiology Experimental5(1), 40 (2021)
2021
-
[22]
arXiv preprint arXiv:1802.03426 (2018)
McInnes, L., Healy, J., Melville, J.: UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
Pith/arXiv arXiv 2018
-
[23]
Nature616, 259–265 (2023)
Moor, M., Banerjee, O., Abad, Z.S.H., et al.: Foundation models for generalist medical artificial intelligence. Nature616, 259–265 (2023)
2023
-
[24]
Academic Radiology19(2), 236–248 (2012)
Moreira, I.C., Amaral, I., Domingues, I., et al.: INbreast: Toward a full-field digital mammographic database. Academic Radiology19(2), 236–248 (2012)
2012
-
[25]
arXiv preprint arXiv:2405.01469 (2024)
Moutakanni, T., Bojanowski, P., Chassagnon, G., et al.: Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning. arXiv preprint arXiv:2405.01469 (2024)
Pith/arXiv arXiv 2024
-
[26]
Scientific Data10(1), 277 (2023)
Nguyen, H.T., Nguyen, H.Q., Pham, H.H., et al.: VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data10(1), 277 (2023)
2023
-
[27]
arXiv preprint arXiv:2304.07193 (2023)
Oquab, M., Darcet, T., Moutakanni, T., et al.: DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
Pith/arXiv arXiv 2023
-
[28]
Biomedical Engineering Letters14(2), 317–330 (2024)
Oza, P., Oza, U., Oza, R., et al.: Digital mammography dataset for breast can- cer diagnosis research (DMID) with breast mass segmentation analysis. Biomedical Engineering Letters14(2), 317–330 (2024)
2024
-
[29]
Radiology314(2), e240597 (2025)
Paschali, M., Chen, Z., Blankemeier, L., et al.: Foundation models in radiology: What, how, why, and why not. Radiology314(2), e240597 (2025)
2025
-
[30]
Nature Machine Intelligence7(1), 119–130 (2025)
Pérez-García, F., Sharma, H., Bond-Taylor, S., et al.: Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence7(1), 119–130 (2025)
2025
-
[31]
Radiological Society of North America: RSNA Screening Mammography Breast Cancer Detection AI Challenge,https://www.kaggle.com/competitions/ rsna-breast-cancer-detection, last accessed 2026/06/30
2026
-
[32]
arXiv preprint arXiv:2507.05201 (2025)
Sellergren, A., Kazemzadeh, S., Jaroensri, T., et al.: MedGemma Technical Report. arXiv preprint arXiv:2507.05201 (2025)
Pith/arXiv arXiv 2025
-
[33]
arXiv preprint arXiv:2508.10104 (2025)
Siméoni, O., Vo, H.V., Seitzer, M., et al.: DINOv3. arXiv preprint arXiv:2508.10104 (2025)
Pith/arXiv arXiv 2025
-
[34]
Apollo - University of Cambridge Repository (2015)
Suckling, J., Parker, J., Dance, D., Astley, S., et al.: Mammographic Image Analysis Society (MIAS) database v1.21. Apollo - University of Cambridge Repository (2015)
2015
-
[35]
arXiv preprint arXiv:2303.00915 (2023) 11
Zhang, S., Xu, Y., Usuyama, N., et al.: BiomedCLIP: A multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915 (2023) 11
Pith/arXiv arXiv 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.