Pith. sign in

REVIEW 2 major objections 5 minor 35 references

Mammography foundation-model transfer is not explained by mammography exposure alone; dataset-level out-of-distribution evaluation is the decisive test.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-14 12:20 UTC pith:IX43H6FT

load-bearing objection Solid multi-site mammography FM benchmark: VLMs lead mean OOD, mammography exposure alone does not, DINOv3 stays hard to beat, and per-dataset heterogeneity is the real takeaway. the 2 major comments →

arxiv 2607.10358 v1 pith:IX43H6FT submitted 2026-07-11 cs.CV cs.AI

Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift

classification cs.CV cs.AI
keywords MammographyFoundation modelsOut-of-distribution generalizationDomain shiftVision-language modelsLinear probeBreast densityBI-RADS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether pretrained foundation models used as frozen image feature extractors for mammography still work when images come from other hospitals, countries, and acquisition setups. Fifteen backbones are compared on three clinical tasks—breast density, BI-RADS assessment, and cancer status—by training only a linear classifier on one source dataset and testing it on many external ones after labels are harmonized. Mammography-specific vision-language models achieve the strongest average external scores, yet simply pretraining or adapting on mammography data does not reliably improve transfer, and a strong natural-image vision model remains hard to beat. Even the leading models win on some external sets and lag on others, and useful features can keep clinical signal while still encoding dataset and scanner structure. The practical claim is that mean scores hide this unevenness, so clinical usefulness of a representation should be judged at the dataset level, not only by in-distribution accuracy or by how much mammography the model has seen.

Core claim

Under a single frozen-backbone linear-probe protocol spanning density, BI-RADS, and cancer status, mammography-specific vision-language models—especially Mammo-FM and MaMA—achieve the highest mean out-of-distribution AUROC, but that advantage is not explained by mammography exposure alone: natural-image DINOv3 stays competitive, single-source mammography adaptation often fails to help, and even the best models show large, task-dependent swings across individual external datasets. Useful representations can preserve clinical signal while retaining dataset and acquisition structure, so dataset-level OOD evaluation—not source performance or pretraining domain alone—is the central criterion for

What carries the argument

A unified frozen-backbone linear-probe benchmark: fifteen foundation encoders are frozen; a task-specific linear head is trained only on one of three source datasets (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer); evaluation uses macro or binary AUROC on twelve task-compatible external datasets after label harmonization, with exam-level max pooling for BI-RADS and cancer. This isolates representation quality under domain shift without confounding full fine-tuning.

Load-bearing premise

The study treats frozen linear-probe AUROC after cross-dataset label harmonization as a fair proxy for representation quality under real domain shift, even though cancer labels mix different clinical definitions and the protocol does not measure the best performance full fine-tuning could reach.

What would settle it

If full fine-tuning on the same sources reversed the model ranking and made mammography-adapted vision-only models match or beat the mammography vision-language models with uniform gains across external datasets—including film and contrast-enhanced sets—then the claim that pretraining type and dataset-level OOD evaluation, rather than mammography exposure or source scores, are the decisive criteria would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Mean OOD AUROC is an incomplete report card; model selection for deployment should report per-dataset scores.
  • Choosing a backbone solely because it was pretrained on mammography can mislead; language alignment and pretraining diversity matter more than modality exposure alone.
  • Strong natural-image self-supervised encoders remain necessary baselines in mammography representation work.
  • Feature-space domain structure can coexist with clinical utility; full domain invariance is not required for useful transfer.
  • Claims about mammography foundation models should include multi-country, multi-vendor OOD panels rather than single held-out splits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Linear probes may understate how much adaptation could close dataset-specific gaps; full fine-tuning stratified by acquisition type (film, contrast-enhanced, tomosynthesis) would test whether rankings stabilize.
  • Heterogeneous cancer-status definitions across datasets may make some OOD swings partly definitional; standardizing binary cancer proxies before ranking would isolate true representation transfer.
  • If metadata-derived text (density, view) drives MaMA’s density edge while clinical-report text drives Mammo-FM’s BI-RADS and cancer edge, pretraining text design becomes a controllable lever for task-targeted transfer.
  • Multi-site prospective screening cohorts would be the natural next stress test beyond public archives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper benchmarks 15 foundation-model backbones as frozen mammography feature extractors under a unified linear-probe protocol. Using three source datasets (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer) and up to 12 task-compatible OOD datasets after label harmonization, it compares natural-image SSL, general radiology SSL, mammography-adapted SSL, general medical VLMs, and mammography-specific VLMs on density, BI-RADS, and cancer status. The central claims are that mammography-specific VLMs (especially Mammo-FM and MaMA) achieve the strongest mean OOD AUROC, that mammography exposure alone does not explain robustness (DINOv3 remains competitive; single-source adaptation can hurt), and that even leading models show heterogeneous per-dataset OOD performance, so dataset-level OOD evaluation should be a primary criterion. Supporting analyses include ID–OOD scatter, family averages, per-dataset heatmaps, and UMAP feature inspection.

Significance. If the comparative ranking holds under the stated protocol, the paper provides a timely and practically useful reference for mammography representation learning. The multi-task, multi-country design, explicit leakage control (RSNA-site2 only), public code, and honest limitation that linear probes measure representation quality rather than best absolute performance are real strengths. The finding that mammography-specific VLMs lead while DINOv3 remains hard to beat, and that continued pretraining on a single mammography corpus does not reliably improve OOD transfer, is actionable for model selection and for future pretraining design. Dataset-level heterogeneity and feature-space structure further push the field beyond single-number leaderboards.

major comments (2)
  1. §3.1 and Table 1: cancer-status OOD averages mix heterogeneous label definitions (BI-RADS-derived positives in VinDr, screening vs cancer-enriched cohorts, coarse normal/abnormal labels). The paper acknowledges this and reports a weak ID–OOD rank correlation (ρ=0.61) and a slightly positive mean gap, but still presents mean cancer OOD AUROC as a primary ranking axis in Table 3 and the abstract. Please either (i) restrict the cancer mean to a clearly defined subset of comparable targets, or (ii) demote cancer mean OOD to a secondary/exploratory metric and lead with density and BI-RADS, so the headline ranking is not driven by an averaged mixture of different tasks.
  2. §2.2 and §3.3: several mammography-specific models (MaMA, GLAM, VersaMammo, Mammo-CLIP, Mammo-FM) report pretraining exposure to EMBED, VinDr-Mammo, and/or RSNA—the same sources used as ID training sets. The protocol freezes the backbone and trains only a linear head, which limits direct leakage, but partial source overlap still weakens the claim that mammography exposure alone does not explain robustness and that OOD gains are purely from objective/alignment. Please add an explicit overlap table (model × source datasets) and, where possible, a leave-source-out or non-overlapping OOD subset analysis so the family comparison is not confounded by shared pretraining data.
minor comments (5)
  1. Table 1: the † exclusion rule for incomplete BI-RADS ordinal targets is stated in the caption but not fully reflected in the main text of §2.1; a short sentence listing which datasets are dropped from the final BI-RADS OOD mean would help.
  2. Figure 2: the zoom insets are useful but the error bars are hard to read at the printed scale; consider larger markers or a supplementary table of exact ID/OOD values with CIs.
  3. §2.3: exam-level max pooling of image probabilities is a reasonable clinical aggregation, but a brief sensitivity check (mean pooling or majority) would strengthen the claim that rankings are not aggregation-dependent.
  4. §3.5 / Figure 4: UMAP is qualitative; if space allows, a simple linear probe or silhouette score for density/BI-RADS vs manufacturer/view would make the “clinical signal retained while domain structure remains” claim more quantitative.
  5. Typos / consistency: “Ray DINO” vs “RayDINO” in Table 3; “Mammo-FMandMaMA” spacing in the abstract; arXiv IDs and access dates for challenge datasets should be standardized in the references.

Circularity Check

0 steps flagged

No circular derivation: comparative frozen linear-probe OOD benchmark on external public datasets, not a first-principles result forced by its inputs.

full rationale

This paper is an empirical multi-dataset benchmark, not a derivation that claims to predict or prove a quantity from first principles. Fifteen externally pretrained backbones are frozen; task heads are fit only on designated source splits (VinDr-Mammo for BI-RADS, EMBED for density, RSNA-site2 for cancer) and scored with AUROC on held-out ID and task-compatible OOD sets after label harmonization (Tables 1–3, Figures 2–3). Mean OOD rankings, family averages, and the claim that mammography exposure alone does not explain robustness are measurements under that protocol, not quantities recovered by construction from fitted constants. Pretraining-source overlap (e.g., MaMA/GLAM/Mammo-FM/VersaMammo seeing EMBED or VinDr during pretraining) and cancer-label heterogeneity are methodological confounds the paper partly acknowledges (§2.2, §3.1, §4); they do not make the reported OOD AUROCs tautological with the evaluation targets. Self-citations (e.g., Glocker et al. on chest FM bias) are background, not load-bearing uniqueness or ansatz imports. Feature UMAPs are qualitative inspection, not circular proofs. Per the stated criteria, there is no self-definitional step, no fitted input renamed as prediction, and no self-citation chain that forces the central comparative claims. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper’s load-bearing content is empirical protocol design, not new physical entities or free-form theory. The central claims rest on standard ML evaluation assumptions plus several domain choices about labels, aggregation, and what frozen linear probes measure. No new particles/forces are introduced; free parameters are protocol knobs rather than fitted physical constants.

free parameters (4)
  • image resize to 1024×768
    All images are resized to a fixed resolution preserving global breast context; this hand-chosen input size can favor some architectures/pretraining resolutions over others.
  • exam-level max-pool aggregation of image probabilities
    BI-RADS and cancer image probabilities are aggregated by per-class max pooling then renormalization; this is a design choice that defines the exam-level score being ranked.
  • source-dataset assignment per task
    VinDr for BI-RADS, EMBED for density, RSNA-site2 for cancer are chosen as ID sources; rankings and gaps are conditional on these training domains.
  • 1000 bootstrap resamples for 95% CIs
    Uncertainty half-widths depend on this bootstrap procedure and on which OOD datasets enter each mean.
axioms (5)
  • domain assumption Frozen-backbone linear probes measure transferable representation quality under domain shift.
    Stated throughout §2.3 and used as the sole adaptation protocol; §4 admits this does not equal best absolute task performance under full fine-tuning.
  • domain assumption Cross-dataset label harmonization yields comparable density, BI-RADS, and cancer tasks.
    Required by Table 1 filtering and §2.1; cancer especially mixes BI-RADS-derived, screening, and enriched cohort definitions (§3.1).
  • domain assumption Macro one-vs-rest AUROC (density/BI-RADS) and binary AUROC (cancer) are appropriate primary metrics for ranking models under class imbalance and ordinal labels.
    §2.3 defines these as primary metrics; ordinal BI-RADS structure is not modeled beyond multiclass AUROC.
  • domain assumption Public pretrained checkpoints can be compared as fixed feature extractors without re-pretraining under matched compute/data budgets.
    §2.2 uses released models as-is except three EMBED-adapted SSL variants; confounds pretraining scale, data access, and objective.
  • standard math Standard bootstrap confidence intervals and mean OOD averages over task-compatible datasets with finite AUROC are valid summaries.
    §2.3; the paper partially mitigates average-only bias via dataset-level plots in §3.4.

pith-pipeline@v1.1.0-grok45 · 15129 in / 3555 out tokens · 45382 ms · 2026-07-14T12:20:43.034337+00:00 · methodology

0 comments
read the original abstract

Foundation models are increasingly used as image feature extractors for mammography, but their robustness under external domain shift remains unclear. We benchmark 15 foundation-model backbones across breast density, BI-RADS severity, and cancer status using a unified frozen-backbone linear-probe protocol, training on 3 source datasets and evaluating on 12 task-compatible out-of-distribution (OOD) datasets after label harmonization. Mammography-specific vision-language models (Mammo-FM and MaMA) provide the strongest mean OOD performance, but robustness is not explained by mammography exposure alone. DINOv3 remains a competitive vision-only baseline, and mammography-adapted pretraining does not consistently improve generalization. Dataset-level analysis further shows that even leading models show heterogeneous performance across datasets. Feature-space inspection reveals that useful representations can preserve clinical signal while retaining dataset and acquisition structure. These findings highlight dataset-level OOD evaluation as a central criterion for assessing mammography representations. Our code is publicly available: https://github.com/biomedia-mira/mammo-ood.

Figures

Figures reproduced from arXiv: 2607.10358 by Ben Glocker, Emma A.M. Stanley, Giang Nguyen, Hieu Pham, Raghav Mehta, Thi Hao Nguyen, Tian Xia.

Figure 1
Figure 1. Figure 1: Overview of the mammography foundation-model benchmark. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: ID-to-OOD generalization of foundation-model representations. Each point is [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dataset-level AUROC performances (with 95% confidence interval error-bar) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: UMAP inspection grid for five representative backbones. Rows are (top to bot [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 12 linked inside Pith

  1. [1]

    Data6(11), 111 (2021)

    Alsolami, A.S., Shalash, W., Alsaggaf, W., et al.: King Abdulaziz University Breast Cancer Mammogram Dataset (KAU-BCMD). Data6(11), 111 (2021)

  2. [2]

    American College of Radiology: ACR BI-RADS Atlas: Breast Imaging Reporting and Data System. 5th edn. American College of Radiology, Reston (2013)

  3. [3]

    Data in Brief55, 110633 (2024)

    Aqdar, K.B., Mustafa, R.K., Abdulqadir, Z.H., et al.: Mammogram mastery: A robust dataset for breast cancer detection and medical education. Data in Brief55, 110633 (2024)

  4. [4]

    arXiv preprint arXiv:2108.07258 (2021)

    Bommasani, R., Hudson, D.A., Adeli, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

  5. [5]

    JAMA Network Open4(8), e2119100 (2021)

    Buda, M., Saha, A., Walsh, R., et al.: A data set and deep learning algorithm for the detection of masses and architectural distortions in digital breast tomosynthesis images. JAMA Network Open4(8), e2119100 (2021)

  6. [6]

    Scientific Data10(1), 123 (2023)

    Cai, H., Wang, J., Dan, T., et al.: An online mammography database with biopsy confirmed types. Scientific Data10(1), 123 (2023)

  7. [7]

    arXiv preprint arXiv:2409.18119 (2024) 9

    Du, Y., Onofrey, J., Dvornek, N.C.: Multi-view and multi-scale alignment for contrastive language-image pre-training in mammography. arXiv preprint arXiv:2409.18119 (2024) 9

  8. [8]

    arXiv preprint arXiv:2509.10344 (2025)

    Du, Y., Chen, L., Dvornek, N.C.: GLAM: Geometry-guided local alignment for multi-view VLP in mammography. arXiv preprint arXiv:2509.10344 (2025)

  9. [9]

    arXiv preprint arXiv:2505.10579 (2025)

    Germani, E., Türk, I.S., Zeineddine, F., Mourad, C., Albarqouni, S.: Bias and generalizability of foundation models across datasets in breast mammography. arXiv preprint arXiv:2505.10579 (2025)

  10. [10]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, pp

    Ghosh, S., Poynton, C.B., Visweswaran, S., Batmanghelich, K.: Mammo-CLIP: A vision language foundation model to enhance data efficiency and robustness in mammography. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, pp. 632–642. Springer Nature Switzerland, Cham (2024)

  11. [11]

    arXiv preprint arXiv:2512.00198 (2025)

    Ghosh, S., Joshi, V.P., Syed, R., et al.: Mammo-FM: Breast-specific foundational model for integrated mammographic diagnosis, prognosis, and reporting. arXiv preprint arXiv:2512.00198 (2025)

  12. [12]

    Radiology: Artificial Intelligence5(6), e230060 (2023)

    Glocker, B., Jones, C., Roschewitz, M., Winzeck, S.: Riskofbiasinchestradiography deep learning foundation models. Radiology: Artificial Intelligence5(6), e230060 (2023)

  13. [13]

    Radiology: Artificial Intelligence3(1), e200103 (2021)

    Halling-Brown, M.D., Warren, L.M., Ward, D., et al.: OPTIMAM Mammography Image Database: A large-scale resource of mammography images and clinical data. Radiology: Artificial Intelligence3(1), e200103 (2021)

  14. [14]

    In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    He, K., Chen, X., Xie, S., et al.: Masked autoencoders are scalable vision learners. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979–15988. IEEE (2022)

  15. [15]

    arXiv preprint arXiv:2509.20271 (2025)

    Huang, F., Zhu, J., Yu, Y., et al.: A versatile foundation model for AI-enabled mammogram interpretation. arXiv preprint arXiv:2509.20271 (2025)

  16. [16]

    Radiology: Artificial Intelligence5(1), e220047 (2023)

    Jeong, J.J., Vey, B.L., Bhimireddy, A., et al.: The EMory BrEast imaging Dataset (EMBED):Araciallydiverse, granulardatasetof3.4millionscreeninganddiagnostic mammographic images. Radiology: Artificial Intelligence5(1), e220047 (2023)

  17. [17]

    Scientific Data12(1), 1479 (2025)

    Kendall, E., Hajishafiezahramini, P., Hamilton, M., et al.: Full field digital mam- mography dataset from a population screening program. Scientific Data12(1), 1479 (2025)

  18. [18]

    Scientific Data 9(1), 122 (2022)

    Khaled, R., Helal, M., Alfarghaly, O., et al.: Categorized contrast enhanced mam- mography dataset for diagnostic and artificial intelligence research. Scientific Data 9(1), 122 (2022)

  19. [19]

    arXiv preprint arXiv:2412.10372 (2024)

    Khattak, M.U., Kunhimon, S., Naseer, M., Khan, S., Khan, F.S.: UniMed-CLIP: To- wards a unified image-text pretraining paradigm for diverse medical imaging modal- ities. arXiv preprint arXiv:2412.10372 (2024)

  20. [20]

    Scientific Data 4(1), 170177 (2017) 10

    Lee, R.S., Gimenez, F., Hoogi, A., Miyake, K.K., et al.: A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific Data 4(1), 170177 (2017) 10

  21. [21]

    European Radiology Experimental5(1), 40 (2021)

    Loizidou, K., Skouroumouni, G., Pitris, C., Nikolaou, C.: Digital subtraction of temporally sequential mammograms for improved detection and classification of mi- crocalcifications. European Radiology Experimental5(1), 40 (2021)

  22. [22]

    arXiv preprint arXiv:1802.03426 (2018)

    McInnes, L., Healy, J., Melville, J.: UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)

  23. [23]

    Nature616, 259–265 (2023)

    Moor, M., Banerjee, O., Abad, Z.S.H., et al.: Foundation models for generalist medical artificial intelligence. Nature616, 259–265 (2023)

  24. [24]

    Academic Radiology19(2), 236–248 (2012)

    Moreira, I.C., Amaral, I., Domingues, I., et al.: INbreast: Toward a full-field digital mammographic database. Academic Radiology19(2), 236–248 (2012)

  25. [25]

    arXiv preprint arXiv:2405.01469 (2024)

    Moutakanni, T., Bojanowski, P., Chassagnon, G., et al.: Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning. arXiv preprint arXiv:2405.01469 (2024)

  26. [26]

    Scientific Data10(1), 277 (2023)

    Nguyen, H.T., Nguyen, H.Q., Pham, H.H., et al.: VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data10(1), 277 (2023)

  27. [27]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., et al.: DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

  28. [28]

    Biomedical Engineering Letters14(2), 317–330 (2024)

    Oza, P., Oza, U., Oza, R., et al.: Digital mammography dataset for breast can- cer diagnosis research (DMID) with breast mass segmentation analysis. Biomedical Engineering Letters14(2), 317–330 (2024)

  29. [29]

    Radiology314(2), e240597 (2025)

    Paschali, M., Chen, Z., Blankemeier, L., et al.: Foundation models in radiology: What, how, why, and why not. Radiology314(2), e240597 (2025)

  30. [30]

    Nature Machine Intelligence7(1), 119–130 (2025)

    Pérez-García, F., Sharma, H., Bond-Taylor, S., et al.: Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence7(1), 119–130 (2025)

  31. [31]

    Radiological Society of North America: RSNA Screening Mammography Breast Cancer Detection AI Challenge,https://www.kaggle.com/competitions/ rsna-breast-cancer-detection, last accessed 2026/06/30

  32. [32]

    arXiv preprint arXiv:2507.05201 (2025)

    Sellergren, A., Kazemzadeh, S., Jaroensri, T., et al.: MedGemma Technical Report. arXiv preprint arXiv:2507.05201 (2025)

  33. [33]

    arXiv preprint arXiv:2508.10104 (2025)

    Siméoni, O., Vo, H.V., Seitzer, M., et al.: DINOv3. arXiv preprint arXiv:2508.10104 (2025)

  34. [34]

    Apollo - University of Cambridge Repository (2015)

    Suckling, J., Parker, J., Dance, D., Astley, S., et al.: Mammographic Image Analysis Society (MIAS) database v1.21. Apollo - University of Cambridge Repository (2015)

  35. [35]

    arXiv preprint arXiv:2303.00915 (2023) 11

    Zhang, S., Xu, Y., Usuyama, N., et al.: BiomedCLIP: A multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915 (2023) 11