Pith. sign in

REVIEW 3 major objections 5 minor 19 references

Medical foundation models tested on African brain MRI show no consistent performance gap versus high-resource data when sample sizes are matched, suggesting the main barrier to equitable deployment is data scarcity, not inherent model bias.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Medical foundation models show no consistent generalization gap on African brain MRI; performance differences track dataset size, not data origin.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A candid, useful first benchmark of four medical FMs on African brain MRI; the headline conclusion outruns the evidence, but the paper is worth engaging. the 3 major comments →

arxiv 2607.28771 v1 pith:2BQEGVDD submitted 2026-07-30 cs.CV

Do Medical Foundation Models Generalize on the African Brain?

Kaouther Mouheb , Gonzalo Esteban Mosquera Rojas , Juancito van Leeuwen , Stefan Klein , Esther E. Bron This is my paper
classification cs.CV
keywords medical foundation modelsbrain MRIAfrican neuroimaginggeneralizationtumor segmentationdementia classificationdataset biasBraTS-Africa
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether medical foundation models trained largely on European and North American brain MRI also work on African brain scans. Across two tasks — dementia classification on a Nigerian dataset and tumor segmentation on BraTS-Africa — four foundation models are compared with from-scratch baselines and with size-matched non-African datasets. The central claim is that performance gaps between African and non-African cohorts are inconsistent in direction and mostly track training set size, not population origin. If true, the main obstacle to equitable medical AI in Africa is not hidden model bias but the scarcity and limited diversity of publicly available African neuroimaging data.

Core claim

The authors aim to show that on the tested tasks, medical foundation models generalize to African brain MRI as well as they do to high-resource data, and that apparent gaps disappear or reverse when sample size is controlled. For tumor segmentation, MedSAM2 reaches 0.86 Dice on BraTS-Africa versus 0.82 on the size-matched EGD-150, and 3DINO reaches 0.73 versus 0.78. For classification, differences between African and non-African datasets vary up to 9% ROC-AUC and are not consistently in one direction. The paper concludes that the tested FMs do not exhibit an inherent bias against African cohorts and that limited availability of African datasets is the main barrier to robust evaluation and de

What carries the argument

The load-bearing device is the size-matched comparison: a randomly drawn 150-case subset of the Dutch Erasmus Glioma Dataset (EGD-150) is paired against the 146-case BraTS-Africa dataset so that training sample size is equal in each Monte-Carlo split. Four pretrained models are the test instruments: two generalist encoders (BrainIAC, 3DINO) and two promptable segmentation models (MedSAM2, Medical-SAM2). The size matching is what lets the authors attribute cross-population differences to data origin rather than to the amount of training data.

Load-bearing premise

The conclusion relies on the premise that equating sample size between BraTS-Africa and EGD-150 leaves population origin as the only meaningful difference, even though the datasets also differ in tumor-type composition (BraTS-Africa includes 51 non-glioma cases), acquisition protocols, and image quality.

What would settle it

A controlled comparison on two datasets matched for sample size, tumor-type composition, acquisition protocol, and image quality would settle the claim: if a consistent performance drop still appears on the African cohort across all four models, the absence-of-bias conclusion fails. Alternatively, a re-analysis of BraTS-Africa restricted to the 95 gliomas, compared with glioma-only EGD-150, is a concrete check a reader could run.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Medical foundation models can be used productively on African brain MRI without additional pretraining; the strongest gains appear in low-data settings.
  • For tumor segmentation, promptable SAM-based models can perform close to supervised baselines zero-shot, which matters where annotated African data are scarce.
  • Fine-tuning generalist encoders with a small African dataset yields only modest gains, so the practical value of FMs in this setting is greatest when data are very limited.
  • As larger African datasets become available, from-scratch trained models catch up to FMs, so the current advantage of FMs on African data is partly a data-scarcity artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If a large, diverse African neuroimaging dataset were collected, the paper's logic predicts that the gap between FMs and from-scratch models would narrow, and any remaining gap would need another explanation.
  • A testable extension: measure FM performance on subgroup-defined African cohorts (e.g., separated by tumor subtype) to see whether non-glioma cases explain part of the BraTS-Africa versus EGD gap.
  • The paper's age-confounding caveat on the Nigerian classification dataset could be tested by matching age between dementia and control groups and re-running the same probes.
  • Because the highest FM gains appear in zero-shot segmentation, a practical deployment path is to use promptable segmentation FMs with clinician-provided boxes in African centers, bypassing fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates four medical foundation models (BrainIAC, 3DINO, MedSAM2, Medical-SAM2) on African brain MRI across two tasks: dementia classification (Nigerian Brain dataset, n=50, vs OASIS-4, n=97) and brain tumor segmentation (BraTS-Africa, n=146, vs a size-matched subset of the Dutch EGD, n=150). Generalist FMs are adapted via linear probing/LoRA for classification and with a UNETR decoder for segmentation; segmentation-specific FMs are tested zero-shot with ground-truth bounding-box prompts. The paper reports that FMs provide limited gains in classification, consistently improve segmentation over a from-scratch U-Net, and that African/non-African performance differences are inconsistent and appear more related to dataset size than data origin. It concludes that FMs do not exhibit an inherent bias against African cohorts and that the main barrier is scarcity of African datasets.

Significance. If the central claim were fully supported, this would be a valuable contribution to the underexplored area of medical FM evaluation in African populations. Strengths include the use of public African datasets, evaluation of four diverse FMs, a size-matched comparison for segmentation, per-case distribution figures, and public code. The authors also transparently acknowledge several limitations, including small sample sizes, an age confounder in the classification data, and oracle prompting. However, the headline conclusion is under-supported because the main African/non-African comparison is confounded by tumor-type composition and acquisition differences, and the statistical evidence is only descriptive. The paper is a useful benchmark but needs substantial revision to align its claims with the evidence.

major comments (3)
  1. [§2.1 / Table 2 / §4] The size-matched comparison between BraTS-Africa and EGD-150 does not control for tumor-type composition: BraTS-Africa contains 51 non-glioma cases while EGD is a glioma dataset. The union of sub-region masks does not make the tasks equivalent, as lesion morphology and boundary characteristics differ across tumor types. Consequently, the MedSAM2 advantage on BraTS-Africa (0.86 vs 0.82, Table 2) and the conclusion that the high-resource advantage is 'driven primarily by sample size' (Section 4) rest on an uncontrolled confound. Please report a glioma-only subset analysis or stratify by tumor type, and temper the sample-size claim.
  2. [§2.1, Table 1, §4] The classification comparison is not size-matched: Nigerian Brain has n=50 while OASIS-4 has n=97, and Section 4 admits a substantial age gap between dementia and healthy groups in the Nigerian dataset that may drive performance via age-related features. The Discussion claims to compare 'under matched sample-size', which is false for classification. The classification results therefore cannot support the conclusion that FMs show no inherent bias. Please either match sample size, adjust for age, or explicitly exclude classification from the matched comparison.
  3. [§2.3 / §3 / §4] The study uses only 5 Monte-Carlo splits with descriptive statistics and no formal hypothesis or equivalence tests. Many reported differences (e.g., BrainIAC linear probing 0.85±0.06 vs 3DINO 0.80±0.03 on Nigerian; MedSAM2 0.86±0.01 vs Medical-SAM2 0.62±0.01 on BraTS-Africa) are large, but the claim that performance differences are 'inconsistent' across populations is based on overlapping distributions. The abstract's statement that FMs 'do not exhibit an inherent bias' overstates the evidence; the study can only say that no consistent gap was detected in these small, confounded datasets. Please soften the conclusion or provide formal tests (e.g., confidence intervals for the difference) to support strong claims.
minor comments (5)
  1. [Throughout] Typo: 'generalistic' should be 'generalist' (e.g., first paragraph of Section 3.1 and Section 2.3).
  2. [Table 1] The row label 'Fine-tuning' could be clarified as 'LoRA fine-tuning' to match the methods section.
  3. [§2.3] The repository link 'anonymous.4open.science' is not a persistent identifier. Include a DOI or institutional archival link in the published version.
  4. [Figure 1] The per-case Dice distributions for the generalist FMs (Table 3) are not shown in any figure; adding a similar plot would help visualize the spread behind the summary statistics.
  5. [§4] The Discussion mentions that FM gains are largest 'in the low-data regime', but this is not quantified across tasks. Consider reporting a more explicit comparison of FM vs baseline gains at different training set sizes (e.g., EGD-150 vs EGD-large).

Circularity Check

0 steps flagged

No significant circularity: conclusions are empirical, and the acknowledged confounders are validity threats rather than definitional reductions.

full rationale

The paper makes no prediction that is equivalent to a fitted input by construction. It evaluates four externally pre-trained foundation models (none reported to have been trained on the test datasets) and a from-scratch DenseNet121/3D U-Net baseline under 5-fold Monte-Carlo cross-validation; the reported ROC-AUC and Dice values are measured outcomes, not parameters derived from the datasets and then renamed as predictions. The central claim that performance differences 'appear more related to dataset size than data origin' is an inductive interpretation of two contrasts: the size-matched BraTS-Africa (n=146) vs EGD-150 comparison, and the EGD-150 vs EGD-large comparison. That inference is not forced by definition: sample-size matching removes one confounder in the African/non-African contrast, but tumor composition, acquisition protocols, image quality, and age differ between datasets, and the paper itself concedes these limitations. Specifically, Section 4 states that 'the African datasets used are small ... which limits the strength of comparative claims' and that the age gap in the Nigerian dataset 'rais[es] the possibility that the models rely in part on age-related features rather than pathology-specific ones.' Those are external-validity and confounding concerns, not circularity. The only notable institutional overlap is the use of the Erasmus Glioma Dataset from the authors' center, but EGD is an external public benchmark used as a data source, not as a derived theorem or fitted result; no load-bearing conclusion is established by citing the authors' own prior work. No self-citation chain, imported uniqueness theorem, or ansatz-by-citation is present. Accordingly, the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No free parameters are fitted in this evaluation; the claims rest on measurement of existing models over public datasets. The three listed axioms are the load-bearing assumptions without which the conclusions would not follow.

axioms (3)
  • domain assumption Sample-size matching between BraTS-Africa and EGD-150 is sufficient to control for data-origin confounders, so any remaining performance gap can be attributed to sample size.
    Underlies the central inference in the Discussion that the advantage of high-resource data is driven by sample size; Section 2.1 selects a size-matched subset for this purpose.
  • domain assumption The ROC-AUC on the Nigerian dataset measures dementia-related signal rather than age-related differences.
    The authors flag a substantial age gap between DE and HC groups; interpreting results as dementia classification assumes this confound does not dominate.
  • domain assumption Ground-truth-derived bounding boxes are a valid proxy for prompts in zero-shot segmentation evaluation.
    All SAM-based segmentation results use tumor bounding boxes from ground truth; the paper notes prompt quality is critical but provides no sensitivity analysis.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Medical Foundation Models Generalize on the African Brain?." pith.science (2026). https://pith.science/paper/2BQEGVDD

@misc{pith2026260728771,
  author       = {Pith},
  title        = {Pith review of: Do Medical Foundation Models Generalize on the African Brain?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BQEGVDD}},
  note         = {Machine review of arXiv:2607.28771}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Medical foundation models (FMs) are increasingly used for brain MRI analysis. However, their evaluation remains dominated by high-resource datasets, leaving generalization to African cohorts underexplored. We assess whether FMs generalize equally to African and non-African brain MRI data across two tasks: dementia classification using a Nigerian dataset and brain tumor segmentation using BraTS-Africa. We evaluate two generalist FMs (BrainIAC, 3DINO) and two segmentation-specific FMs (MedSAM2, Medical-SAM2) against a from-scratch baseline. For classification, FMs provide limited gains (highest ROC-AUC of 0.86 with BrainIAC), whereas for segmentation they consistently improve performance, reaching up to 0.86 Dice with MedSAM2. Performance differences between African and non-African cohorts are inconsistent and appear more related to dataset size than data origin. These results suggest that FMs do not exhibit an inherent bias against African cohorts, and highlight the limited availability and diversity of African neuroimaging datasets as the main barrier to robust evaluation and deployment.

Figures

Figures reproduced from arXiv: 2607.28771 by Esther E. Bron, Gonzalo Esteban Mosquera Rojas, Juancito van Leeuwen, Kaouther Mouheb, Stefan Klein.

Figure 1
Figure 1. Figure 1: Per-case Dice score distributions for zero-shot tumor segmentation on BraTS￾Africa (n = 146) and EGD-150. Combined denotes averaging the output probability maps across all four MRI sequences; FLAIR denotes using the FLAIR sequence only [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ground truth segmentation and model outputs for three example cases from BraTS-Africa. Results are obtained using the FLAIR sequence as input. for all three models, with differences of at most 0.05 DSC. With the larger EGD training set, all models improve substantially, reaching 0.80 ± 0.01 for BrainIAC and 0.82 ± 0.01 for 3DINO. In this setting, both models perform comparably to the from-scratch baseline … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 3 linked inside Pith

  1. [1]

    Ra- diology: Artificial Intelligence7(4), e240528 (2025)

    Adewole, M., Rudie, J.D., Gbadamosi, A., Zhang, D., Raymond, C., Ajigbotoshso, J., Toyobo, O., Aguh, K., Omidiji, O., Akinola, R., et al.: The brats-africa dataset: expanding the brain tumor segmentation data to capture african populations. Ra- diology: Artificial Intelligence7(4), e240528 (2025)

    2025

  2. [2]

    Advances in Neural Information Processing Systems38, 118923–118957 (2026)

    Cuttano, C., Trivigno, G., Averta, G., Masone, C.: Sansa: Unleashing the hid- den semantics in sam2 for few-shot segmentation. Advances in Neural Information Processing Systems38, 118923–118957 (2026)

    2026

  3. [3]

    arXiv preprint arXiv:2508.13626 (2025)

    Danaei, S., Dehghanian, Z., Meftah, E., Naderi, N., Safavi-Naini, S.A.A., Kho- rasanizade, F., Rabiee, H.R.: State of abdominal ct datasets: A critical re- view of bias, clinical relevance, and real-world applicability. arXiv preprint arXiv:2508.13626 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    In: Inter- national workshop on biomedical image registration

    Fischer, B., Modersitzki, J.: Flirt: A flexible image registration toolbox. In: Inter- national workshop on biomedical image registration. pp. 261–270. Springer (2003)

    2003

  5. [5]

    In: Proceedings of the IEEE/CVF winter conference on applications of computer vi- sion

    Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vi- sion. pp. 574–584 (2022)

    2022

  6. [6]

    Iclr1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr1(2), 3 (2022)

    2022

  7. [7]

    Human brain mapping 40(17), 4952–4964 (2019)

    Isensee, F., Schell, M., Pflueger, I., Brugnara, G., Bonekamp, D., Neuberger, U., Wick, A., Schlemmer, H.P., Heiland, S., Wick, W., et al.: Automated brain extrac- tion of multisequence mri using artificial neural networks. Human brain mapping 40(17), 4952–4964 (2019)

    2019

  8. [8]

    arXiv preprint arXiv:2511.09742 (2025)

    Li, F., Dapamede, T., Chavoshi, M., Jeon, Y.S., Khosravi, B., Dere, A., Brown- Mulry, B., Isaac, R.S., Mansuri, A., Sanyika, C., et al.: Feature quality and adapt- ability of medical foundation models: A comparative evaluation for radiographic classification and segmentation. arXiv preprint arXiv:2511.09742 (2025)

    arXiv 2025

  9. [9]

    arXiv preprint arXiv:2504.03600 (2025)

    Ma, J., Yang, Z., Kim, S., Chen, B., Baharoon, M., Fallahpour, A., Asakereh, R., Lyu, H., Wang, B.: Medsam2: Segment anything in 3d medical images and videos. arXiv preprint arXiv:2504.03600 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    IEEE transactions on medical imaging 34(10), 1993–2024 (2014)

    Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34(10), 1993–2024 (2014)

    1993

  11. [11]

    Big Data and Cognitive Computing10(3), 76 (2026)

    Musa, A., Prasad, R., Onwualu, P., Hernandez, M.: A systematic review of cross- population shifts in medical imaging analysis with deep learning. Big Data and Cognitive Computing10(3), 76 (2026)

    2026

  12. [12]

    Radiology314(2), e240597 (2025)

    Paschali, M., Chen, Z., Blankemeier, L., Varma, M., Youssef, A., Bluethgen, C., Langlotz, C., Gatidis, S., Chaudhari, A.: Foundation models in radiology: what, how, why, and why not. Radiology314(2), e240597 (2025)

    2025

  13. [13]

    Nature Neuroscience pp

    Tak, D., Garomsa, B.A., Zapaishchykova, A., Chaunzwa, T.L., Climent Pardo, J.C., Ye, Z., Zielke, J., Ravipati, Y., Pai, S., Vajapeyam, S., et al.: A generalizable foundation model for analysis of human brain mri. Nature Neuroscience pp. 1–12 (2026)

    2026

  14. [14]

    IEEE transactions on medical imag- ing29(6), 1310–1320 (2010) 10 K

    Tustison, N.J., Avants, B.B., Cook, P.A., Zheng, Y., Egan, A., Yushkevich, P.A., Gee, J.C.: N4itk: improved n3 bias correction. IEEE transactions on medical imag- ing29(6), 1310–1320 (2010) 10 K. Mouheb et al

    2010

  15. [15]

    Neuro-oncology25(2), 279–289 (2023)

    van der Voort, S.R., Incekara, F., Wijnenga, M.M., Kapsas, G., Gahrmann, R., Schouten, J.W., Nandoe Tewarie, R., Lycklama, G.J., De Witt Hamer, P.C., Ei- jgelaar, R.S., et al.: Combined molecular subtyping, grading, and segmentation of glioma using multi-task deep learning. Neuro-oncology25(2), 279–289 (2023)

    2023

  16. [16]

    IEEE Transactions on Neural Networks and Learning Systems (2025)

    Wang, H., Guo, S., Ye, J., Deng, Z., Cheng, J., Li, T., Chen, J., Su, Y., Huang, Z., Shen, Y., et al.: Sam-med3d: a vision foundation model for general-purpose seg- mentation on volumetric medical images. IEEE Transactions on Neural Networks and Learning Systems (2025)

    2025

  17. [17]

    Scientific Data12(1), 518 (2025)

    Wogu, E., Filima, P., Caron, B., Deabler, D., Herholz, P., Leal, C., Mehboob, M.F., Kim, S., Gosain, A., Flexwala, A., et al.: A labeled clinical-mri dataset of nigerian brains. Scientific Data12(1), 518 (2025)

    2025

  18. [18]

    npj Digital Medicine8(1), 639 (2025)

    Xu, T., Hosseini, S., Anderson, C., Rinaldi, A., Krishnan, R.G., Martel, A.L., Goubran, M.: A generalizable 3d framework and model for self-supervised learning in medical imaging. npj Digital Medicine8(1), 639 (2025)

    2025

  19. [19]

    arXiv preprint arXiv:2408.00874 (2024)

    Zhu, J., Hamdi, A., Qi, Y., Jin, Y., Wu, J.: Medical sam 2: Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874 (2024)

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.