Cross-Dataset Generalization in Breast MRI Tumor Classification via Class-Wise Dataset Mixing

Hamid Usefi; Mohammad Ali Dadrast

REVIEW 3 major objections 5 minor 15 references

Cross-Dataset Generalization in Breast MRI Tumor Classification via Class-Wise Dataset Mixing

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that breast MRI tumor classifiers trained on multiple public datasets can fail on an independent cohort when the training labels are perfectly correlated with dataset origin, and that class-wise dataset mixing restores exte

desk verdict A clear empirical demonstration of dataset-origin shortcuts in breast MRI, but the headline causal claim about class-wise mixing is undercut by an uncontrolled comparison: the mixed arm has ~25% more training data. read the letter →

arxiv 2607.18678 v1 pith:I53NMSF2 submitted 2026-07-21 cs.CV cs.LG

Mohammad Ali Dadrast , Hamid Usefi This is my paper

classification cs.CVcs.LG

keywords breastMRItumorclassificationdataset-originbiasdomaingeneralizationclass-wisedatasetmixingexternalvalidationshortcutlearningmulti-sourcetraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper shows that a binary breast MRI tumor classifier can appear to work while actually exploiting a dataset-origin shortcut. When all positive training slices come from one institution and all negative slices from another, the model learns source identity as a proxy for the label; on an independent multi-center cohort, accuracy falls to near chance (0.5048–0.5265) even though recall stays above 0.99. The proposed fix, class-wise dataset mixing, ensures each label contains slices from both sources, keeping architecture and training pipeline fixed. Under mixing, external accuracy/F1 rise to 0.8884/0.8994 for EfficientNet-B3 and 0.8463/0.8625 for WaveViT-Small. The paper argues that how public datasets are combined matters as much as backbone choice for reliable breast MRI classification.

What carries the argument

The central mechanism is class-wise dataset mixing: building a training set in which the tumor and non-tumor classes each contain slices from both Duke and fastMRI, so that dataset origin is decorrelated from the label. In this paper, label 1 combines all Duke positive slices with fastMRI positives (up to 40% of the Duke count), and label 0 combines all fastMRI negatives with a like-sized share of Duke negatives. The confounded condition, where label perfectly tracks origin, is the control that exposes the shortcut learning.

What would settle it

A control experiment that trains on a confounded set augmented with extra slices and rebalanced classes while keeping label–origin correlation intact, then compares external accuracy; if accuracy rises without mixing, the mixing label is not the active ingredient. Alternatively, re-running the mixing experiment with the mixed set downsampled to match the confounded set's slice counts and class proportions.

Watch

Extended reading notes

Core claim

The paper's central claim is that dataset-origin bias, not architectural weakness, causes cross-dataset failure in slice-level breast MRI tumor classification. In a deliberately confounded training set where label is perfectly correlated with dataset origin (malignant slices from Duke, negative slices from fastMRI), both EfficientNet-B3 and WaveViT-Small achieve near-chance accuracy on the external MAMA-MIA cohort despite predicting almost everything positive. Replacing the confounded set with a class-wise mixed set, where each class contains slices from both Duke and fastMRI, breaks the shortcut and restores external accuracy to 0.8463–0.8884 and F1 to 0.8625–0.8994, with the same architect

Load-bearing premise

The claim that class-wise mixing is what causes the improvement assumes the confounded and mixed training conditions are otherwise comparable; but the mixed set has about 24% more slices and a different class balance, and the external test slice-level class prevalence is not reported.

Editorial extensions

If this is right

Multi-source training that ignores label–origin correlation can silently produce models with near-chance external accuracy despite high internal and recall performance.
Class-wise mixing is a simple, architecture-agnostic data composition step that can recover external accuracy without changing the model or loss.
Slice-level external validation on a held-out multi-center cohort (MAMA-MIA) is a stringent test for dataset-origin bias in breast MRI classification.
The finding transfers across a convolutional network (EfficientNet-B3) and a wavelet-transformer hybrid (WaveViT-Small), suggesting the effect is data-composition-driven, not architecture-driven.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves open whether the improvement comes strictly from breaking the label–origin correlation or from the larger, differently balanced training set; a matched-count control would isolate the mechanism.
If the mechanism generalizes, the same mixing principle could be applied to other medical imaging domains where classes are unevenly distributed across acquisition sites, such as chest X-ray or histopathology datasets.
A natural extension is patient-level aggregation: since slice-level predictions are highly correlated within a patient, averaging or voting over slices could convert the reported slice-level gains into clinically meaningful per-study decisions.
The authors' method is a form of decorrelation at the data level; a similar effect could be achieved procedurally by explicitly balancing source proportions per class in the loss or via domain-adversarial training, an untested alternative.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the negative control is real and useful: train on Duke positives plus fastMRI negatives, where label is perfectly correlated with source, and both models collapse to near-chance accuracy on MAMA-MIA while predicting almost everything positive. Second, the positive result—class-wise mixing restoring accuracy to 0.85–0.89—is not yet causally secured, because the mixed arm differs from the confounded arm in ways that have nothing to do with decorrelating label and source.

What is actually new is narrow but concrete: the specific quantitative results on these three public breast MRI datasets. The general phenomenon (shortcut learning, dataset bias) is already well established, and the paper cites Zech et al. and the external-validation reviews. The contribution is the demonstration in breast MRI and a simple, repeatable mitigation recipe. That is worth having.

The paper also does several things right. It uses patient-level splitting, keeps MAMA-MIA completely separate from training and tuning, holds the architectures and training pipeline fixed across arms, and is honest about the slice-level nature of the evaluation. The writing is clear and the experiments are easy to follow.

The soft spot is the uncontrolled comparison between the confounded and mixed training sets. From Table II, the mixed training set has 31,199 slices; the confounded set has roughly 24,900. The mixed set also adds fastMRI positives and Duke negatives, so it is not just larger—it is compositionally different. The paper's claim that the gains come from class-wise mixing is therefore not supported. A size-matched confounded control (add extra Duke negatives and fastMRI positives to the confounded arm while keeping label perfectly correlated with source) would isolate the intervention. There is also no sensitivity analysis on the hand-set 40% mixing fraction, which is a free parameter. Those are the load-bearing gaps.

Minor issues: all metrics are slice-level with no per-patient aggregation, no confidence intervals, and the MAMA-MIA test prevalence is not reported. These are secondary for a proof-of-concept, but they should be addressed in a revision.

Who is this for? Researchers combining multiple public medical imaging datasets, especially in breast MRI. It is a cautionary result and a baseline mitigation, not a new method. It deserves a serious referee—the direction is likely right, but the causal claim needs the matched ablation before it can be accepted. I would engage with it.

Referee Report

3 major / 5 minor

Summary. The paper studies cross-dataset generalization in breast MRI tumor classification. The authors train EfficientNet-B3 and WaveViT-Small on two public datasets (Duke and fastMRI) and evaluate strictly on the independent MAMA-MIA cohort. In a deliberately confounded setup, where the positive class consists entirely of Duke slices and the negative class entirely of fastMRI slices, both models achieve near-chance external accuracy (0.5048–0.5265) despite very high recall. They then construct a 'mixed' training set in which each class contains slices from both datasets, retrain the same architectures with the same pipeline, and report substantially improved MAMA-MIA accuracy/F1 (0.8463/0.8625 for WaveViT-Small; 0.8884/0.8994 for EfficientNet-B3). The paper interprets this as evidence that dataset-origin bias, rather than lack of training data or architecture choice, caused the confounded failure, and that class-wise dataset mixing controls this bias.

Significance. The deliberate confound is a well-designed negative control: it cleanly demonstrates that a standard training pipeline can collapse to near-chance external accuracy when label is perfectly correlated with dataset origin, despite high internal-looking recall. If the causal attribution were secured, the result would be a useful, practical warning for multi-source medical imaging research and for reporting external validation. The use of strict external validation on MAMA-MIA is a genuine strength, as is the effort to preserve patient-level splitting and leakage controls. However, the central claim—that class-wise mixing, rather than increased training set size or changed class composition, drives the improvement—is not supported by the current experimental design, because the mixed and confounded training sets differ in multiple confounded dimensions. The paper is short and reads more like a workshop/negative-result report than a full journal article; with additional controlled experiments it could become a solid methodological contribution.

major comments (3)

[§IV-A, §IV-B, Table II] The comparison between the confounded and mixed conditions is not controlled. From Table II, the Mixed training set contains 22,743 positive and 8,456 negative slices (31,199 total), while the Confounded training set contains approximately 19,071 Duke-positive and 6,040 fastMRI-negative slices (25,111 total)—a ~24% increase in training data. The Mixed condition also adds 3,672 fastMRI-positive and 2,416 Duke-negative slices. Thus the improvement on MAMA-MIA could be due to having more training data, to adding slices that are more similar to MAMA-MIA in appearance, or to a different patient-level split, rather than to the intended decorrelation of label and dataset origin. The paper never states that the identical patient-level split was reused across the two conditions, and no control holds total size, class balance, and patient split fixed while varying only the label–origin correlation
[Table I, §IV-B] All external metrics are slice-level, but MAMA-MIA slice-level class prevalence is not reported, and no patient-level aggregation is given. Because slices from the same patient are highly correlated, accuracy and F1 on slices can be dominated by a small number of patients; the near-chance accuracy in the confounded arm and the high recall in the mixed arm are hard to interpret without knowing the external test set's class balance. Please report (a) the number of positive and negative MAMA-MIA slices and patients, (b) patient-level metrics (e.g., majority-vote or probability-averaged accuracy, sensitivity, specificity), and (c) confidence intervals or bootstrap/patient-level error bars. This is load-bearing for the strength of the external validation claim.
[§IV-B] The 40% mixing fraction is a hand-set free parameter, and no sensitivity analysis is provided. The conclusion that class-wise mixing 'reduces dataset-origin bias' depends on this specific value; a reader cannot tell whether the result is robust to, say, 20% or 60% mixing, or whether the chosen value was selected based on the external test set. Additionally, the weighted cross-entropy loss is mentioned but the exact class weights are not given; if the weights differ between the confounded and mixed conditions, the claim that the 'same training pipeline' was used is weakened. Please specify the weights and report a small sweep over the mixing fraction.

minor comments (5)

[§IV-A] The term 'New Dataset' is capitalized inconsistently; use a consistent label (e.g., 'Confounded Dataset') throughout.
[§III-B] The text says 'the same naming and splitting rules were used in all experiments,' but it is not explicitly stated whether the exact patient-level split (80/20) was identical across the confounded and mixed conditions. Please state this explicitly.
[Figs. 2–3] The confusion matrices are referenced but not shown in the text; the figures appear to be omitted or placeholder. Please include the actual matrices or remove the references.
[Table I] No confidence intervals or repeated-run variability are reported. Since training is stochastic and external sets are small, reporting a single run per condition is not sufficient for the claimed precision (e.g., 0.8884).
[References] Minor citation typos: 'V .' in references [3] and [10] should be 'V.'; reference [7] lists 'F. Knoll' in both author and editor positions and should be checked. Ensure consistent citation formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an external-validation experiment with an intentional confound and a mixed-training intervention; no prediction reduces by construction to its input.

full rationale

The paper contains no derivation chain whose output is equivalent to its input by construction. The confounded condition is a deliberately constructed negative control: all positives come from Duke and all negatives from fastMRI, so label is perfectly correlated with dataset origin. The near-chance accuracy on MAMA-MIA is an empirical external-validation result, not an algebraically forced consequence of the construction. The mixed condition is a separate intervention that breaks the label–origin correlation, and both models are evaluated on the same held-out MAMA-MIA cohort. No parameter is fitted to MAMA-MIA, and the 40% mixing proportion is explicitly a hand-set free parameter rather than a quantity fitted to the external test set. The paper's causal interpretation—that class-wise mixing, rather than other differences between training sets, drives the improvement—is a possible threat to experimental validity because the mixed training set has more slices and a different class composition than the confounded set, but this is a correctness/confounding concern, not circularity under the reviewing rules. There are no load-bearing self-citations, no uniqueness theorems, and no ansatz smuggled in via citation. The central experimental logic is self-contained against an external benchmark.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or model entities. Its central result depends on a small set of hand-chosen data-composition parameters and assumptions about label quality and external validity. The 40% mixing ratio is the most explicit free parameter; class weights and slice-selection thresholds are unstated degrees of freedom that affect the metrics.

free parameters (3)

Mixing fraction = 0.4
Chosen by hand in §IV-B to add cross-source variation without letting the smaller source dominate; no sensitivity analysis. The central result depends on this value.
Weighted cross-entropy class weights = unspecified
§III-C says 'weighted cross-entropy loss' but gives no weights; class weights affect the learned decision threshold and external precision/recall.
Slice selection thresholds = Duke first/last 10 slices removed; fastMRI negatives slices 20–170; MAMA-MIA inclusion criteria
§III-B defines these thresholds; they determine the external test set composition (class prevalence), which drives accuracy/F1.

assumptions (4)

domain assumption Slice-level labels derived from Duke bounding boxes, fastMRI mapping files, and MAMA-MIA segmentations are sufficiently accurate for binary tumor classification.
Used throughout §III-B; if label noise differs across datasets, external comparisons are biased.
ad hoc to paper A 40% additive mixture of the minority-source examples is sufficient to break source–label correlation without degrading class balance.
§IV-B; no justification beyond 'adds cross-source variation without letting the smaller source dominate'.
domain assumption Per-slice external accuracy on MAMA-MIA is a meaningful measure of generalization despite strong within-patient slice correlation.
All metrics in Table I are slice-level; authors acknowledge this in §V but do not provide patient-level results.
domain assumption MAMA-MIA's filtered subset (adequate contrast, no artifacts, bilateral, no implants) is representative of external clinical data.
§III-B.2 filters the external cohort, so the reported generalization may not extend to excluded populations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cross-Dataset Generalization in Breast MRI Tumor Classification via Class-Wise Dataset Mixing." pith.science (2026). https://pith.science/paper/I53NMSF2

@misc{pith2026260718678,
  author       = {Pith},
  title        = {Pith review of: Cross-Dataset Generalization in Breast MRI Tumor Classification via Class-Wise Dataset Mixing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I53NMSF2}},
  note         = {Machine review of arXiv:2607.18678}
}

read the original abstract

Breast MRI is highly sensitive for detecting breast tumors, but exams contain many slices and require substantial reading time. Deep learning models often perform well on internal splits but can fail across institutions because of domain shift and dataset-origin bias. We study this failure mode for binary breast MRI tumor classification. EfficientNet-B3 and WaveViT-Small are trained using Duke Breast Cancer MRI and fastMRI, and evaluated only on the independent multi-center MAMA-MIA cohort. In a deliberately confounded setup, where label is perfectly correlated with dataset origin, external accuracy is near chance (0.5048--0.5265), despite very high recall. We then construct a mixed training set in which each class contains samples from both Duke and fastMRI, while preserving patient-level splitting, augmentation, and leakage controls. On MAMA-MIA, dataset mixing improves accuracy/F1 to 0.8463/0.8625 for WaveViT-Small and 0.8884/0.8994 for EfficientNet-B3. These results show that controlling dataset-origin bias is important for reliable breast MRI classification.

Figures

Figures reproduced from arXiv: 2607.18678 by the authors.

**Figure 2.** Confusion matrices for the confounded New Dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Confusion matrices for the Mixed Dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 1 linked inside Pith

[1]

WHO Fact Sheet, updated Apr

World Health Organization.Breast cancer. WHO Fact Sheet, updated Apr. 16, 2026. Available: https://www.who.int/news-room/fact-sheets/ detail/breast-cancer. Accessed Feb. 26, 2026

2026
[2]

L. Sun, R. Legood, I. dos-Santos-Silva, S. M. Gaiha, and Z. Sadique. Global treatment costs of breast cancer by stage: A systematic review. PLOS ONE, 13(11):e0207993, 2018

2018
[3]

A. Saha, M. R. Harowicz, L. J. Grimm, J. Weng, E. H. Cain, C. E. Kim, S. V . Ghate, R. Walsh, and M. A. Mazurowski. Dynamic contrast- enhanced magnetic resonance images of breast cancer patients with tumor locations (Duke-Breast-Cancer-MRI) [Data set].The Cancer Imaging Archive, 2021. doi: 10.7937/TCIA.e3sv-re93

doi:10.7937/tcia.e3sv-re93 2021
[4]

Clark, B

K. Clark, B. Vendt, K. Smith, et al. The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository. Journal of Digital Imaging, 26(6):1045–1057, 2013

2013
[5]

A. Saha, M. R. Harowicz, L. J. Grimm, C. E. Kim, S. V . Ghate, R. Walsh, and M. A. Mazurowski. A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 DCE-MRI features. British Journal of Cancer, 119(4):508–516, 2018

2018
[6]

Zbontar, F

J. Zbontar, F. Knoll, A. Sriram, et al. fastMRI: An open dataset and benchmarks for accelerated MRI.arXiv:1811.08839, 2018

arXiv 2018
[7]

Solomon, P

E. Solomon, P. M. Johnson, Z. Tan, R. Tibrewala, Y . W. Lui, F. Knoll, L. Moy, S. G. Kim, and L. Heacock. FastMRI Breast: A publicly available radial k-space dataset of breast dynamic contrast-enhanced MRI.Radiology: Artificial Intelligence, 7(1):e240345, 2025

2025
[8]

Garrucho, K

L. Garrucho, K. Kushibar, C.-A. Reidel, et al. A large-scale multicenter breast cancer DCE-MRI benchmark dataset with expert segmentations. Scientific Data, 12:453, 2025

2025

Show all 15 references

[9]

Tan and Q

M. Tan and Q. V . Le. EfficientNet: Rethinking model scaling for convolutional neural networks. InProceedings of the 36th International Conference on Machine Learning, PMLR 97, pp. 6105–6114, 2019

2019
[10]

T. Yao, Y . Pan, Y . Li, C.-W. Ngo, and T. Mei. Wave-ViT: Unifying wavelet and transformers for visual representation learning. InComputer Vision – ECCV 2022, LNCS 13685, pp. 328–345, 2022

2022
[11]

J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.PLOS Medicine, 15(11):e1002683, 2018

2018
[12]

A. C. Yu, B. Mohajer, and J. Eng. External validation of deep learning algorithms for radiologic diagnosis: A systematic review.Radiology: Artificial Intelligence, 4(3):e210064, 2022

2022
[13]

C. J. Kelly, A. Karthikesalingam, M. Suleyman, G. Corrado, and D. King. Key challenges for delivering clinical impact with artificial intelligence.BMC Medicine, 17:195, 2019

2019
[14]

G. S. Collins, K. G. M. Moons, P. Dhiman, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods.BMJ, 385:e078378, 2024

2024
[15]

Sounderajah, A

V . Sounderajah, A. Guni, X. Liu, et al. The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nature Medicine, 31:3283–3289, 2025

2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

WHO Fact Sheet, updated Apr

World Health Organization.Breast cancer. WHO Fact Sheet, updated Apr. 16, 2026. Available: https://www.who.int/news-room/fact-sheets/ detail/breast-cancer. Accessed Feb. 26, 2026

2026

[2] [2]

L. Sun, R. Legood, I. dos-Santos-Silva, S. M. Gaiha, and Z. Sadique. Global treatment costs of breast cancer by stage: A systematic review. PLOS ONE, 13(11):e0207993, 2018

2018

[3] [3]

A. Saha, M. R. Harowicz, L. J. Grimm, J. Weng, E. H. Cain, C. E. Kim, S. V . Ghate, R. Walsh, and M. A. Mazurowski. Dynamic contrast- enhanced magnetic resonance images of breast cancer patients with tumor locations (Duke-Breast-Cancer-MRI) [Data set].The Cancer Imaging Archive, 2021. doi: 10.7937/TCIA.e3sv-re93

doi:10.7937/tcia.e3sv-re93 2021

[4] [4]

Clark, B

K. Clark, B. Vendt, K. Smith, et al. The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository. Journal of Digital Imaging, 26(6):1045–1057, 2013

2013

[5] [5]

A. Saha, M. R. Harowicz, L. J. Grimm, C. E. Kim, S. V . Ghate, R. Walsh, and M. A. Mazurowski. A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 DCE-MRI features. British Journal of Cancer, 119(4):508–516, 2018

2018

[6] [6]

Zbontar, F

J. Zbontar, F. Knoll, A. Sriram, et al. fastMRI: An open dataset and benchmarks for accelerated MRI.arXiv:1811.08839, 2018

arXiv 2018

[7] [7]

Solomon, P

E. Solomon, P. M. Johnson, Z. Tan, R. Tibrewala, Y . W. Lui, F. Knoll, L. Moy, S. G. Kim, and L. Heacock. FastMRI Breast: A publicly available radial k-space dataset of breast dynamic contrast-enhanced MRI.Radiology: Artificial Intelligence, 7(1):e240345, 2025

2025

[8] [8]

Garrucho, K

L. Garrucho, K. Kushibar, C.-A. Reidel, et al. A large-scale multicenter breast cancer DCE-MRI benchmark dataset with expert segmentations. Scientific Data, 12:453, 2025

2025