Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection

Aditri Bhagirath; Daniel Frees; Moritz Bolling

REVIEW 3 major objections 5 minor 15 references

Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the best transfer-learning recipe for two small medical-imaging datasets is an ImageNet-pretrained ResNet50 backbone, partially unfrozen, with a 1D convolutional skip-connection head—and that RadImageNet pretraining gi

desk verdict Solid empirical grid search, but the headline AUCs rest on an unstated data-split assumption that could invalidate the rankings. read the letter →

arxiv 2508.17567 v2 pith:TRIKW7EN submitted 2025-08-25 cs.CV cs.LG

Daniel Frees , Moritz Bolling , Aditri Bhagirath This is my paper

classification cs.CVcs.LG

keywords transferlearningconvolutionalneuralnetworksmedicalimageclassificationACLteardetectionbreastlesionmalignancyNetpretrainingRadResNet50

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what a practitioner should actually do when fine-tuning a convolutional network on a small medical imaging dataset, and answers with a concrete recipe tested on two tasks: ACL tear detection from knee MRI and breast nodule malignancy detection from ultrasound. It finds that an ImageNet-pretrained ResNet50 backbone, partially unfrozen, with a lightweight 1D convolutional classifier that has skip connections, produces the best validation AUCs in a broad grid search—0.9969 for ACL tears and 0.9641 for breast malignancy on the test sets. It also finds no evidence that pretraining on RadImageNet, a large radiologist-labeled corpus, beats ImageNet pretraining on either task, directly challenging a prior influential claim. If the result holds, it simplifies transfer-learning choices for medical imaging: generic ImageNet features plus a small residual conv head are a strong default, and domain-specific pretraining should be justified per task rather than assumed.

What carries the argument

The mechanism that drives the result is the fine-tuning setup: take a ResNet50 backbone pre-trained on ImageNet, keep most layers frozen, unfreeze the later blocks, and attach a small classifier made of 1D convolutions with a skip connection. The skip connection acts like a residual block, letting fine-tuning adjust high-level features without destroying the pre-trained representation. Cosine annealing learning-rate decay and light augmentation (flips, rotations, affine transforms) are the supporting controls that stabilize training, and the grid search across backbones, classifier heads, and freezing strategies is what lets the paper attribute gains to these specific choices.

What would settle it

Re-run the best ACL model with a strict exam-level split, placing all slices from one knee exam in only one partition, and compare the test AUC to 0.9969; a large drop would show the original number depended on slice-level leakage. A similar reconstruction for the breast task, grouping images by patient if identifiers are available, would test that result as well.

Watch

Extended reading notes

Core claim

The core claim is that optimal downstream medical classification performance comes from ImageNet-pretrained ResNet50 backbones combined with a 1D convolutional classifier with skip connections, trained with partial backbone unfreezing and cosine annealing learning rate decay. Across a grid search over backbones, classifier heads, and unfreezing strategies, this configuration consistently ranked first on validation AUC, F1, and accuracy for both ACL tear detection and breast nodule malignancy detection. The paper further claims that RadImageNet pretraining did not confer a downstream advantage: ImageNet-initialized models showed better convergence and higher validation AUC in both tasks. Best

Load-bearing premise

The paper treats each MRI slice as an independent sample without stating that every slice from the same knee exam is assigned to the same train/validation/test split; if slices from one exam cross splits, the reported AUCs are inflated by patient-level leakage.

Editorial extensions

If this is right

If the recipe is followed, fine-tuning on small medical datasets can default to an ImageNet-pretrained ResNet50 with a skip-connection conv head rather than heavier or more elaborate architectures.
Domain-specific pretraining corpora should not be assumed superior; task-specific comparison is needed before adopting them.
Partial unfreezing of a ResNet50 backbone beat both fully frozen and fully unfrozen training, making it a strong default schedule.
The grid search rankings by validation AUC mostly matched rankings by F1 and accuracy, so AUC appears to be a reliable selection criterion for these tasks.
The reported test AUCs, 0.9969 for ACL tears and 0.9641 for breast malignancy, are competitive with or better than previously published pipelines on the same tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An editorial test this paper does not run: repeat the best ACL model with all slices from the same knee exam confined to one train/validation/test split; if the 0.9969 AUC holds, the recipe is robust, and if it drops, the headline numbers are partly inflated by slice-level leakage.
The lack of a RadImageNet advantage may stem from these tasks relying on generic edge and texture features that ImageNet already captures; a natural extension is to compare both pretraining corpora on modalities with stronger distribution shift, such as CT or X-ray.
Because no confidence intervals are reported for the best-model AUCs, bootstrap or repeated-seed intervals would clarify whether the architecture differences are real or within run-to-run noise.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here is my honest read. The paper is a careful empirical grid search, not a new method. It does useful things: it compares three classifier heads, three backbones, several unfreezing schemes, and learning-rate schedules on two medical imaging datasets, and it shows box plots of validation AUC/F1/accuracy rather than cherry-picking a single run. The authors also deserve credit for documenting the TensorFlow/PyTorch weight discrepancy in Appendix D; that transparency is rare.

The problems are the ones you flagged. The split is the load-bearing one. MRNet is 1021 exams but the input is per-slice .png images, and the paper never says that slices from the same exam are kept together. If they aren't, the model memorizes patient anatomy and every number in the paper is inflated. The Table 1 typo—'output label for meniscus tear' in the ACL task—makes the bookkeeping look sloppy, and there is no methods section in the version I saw to check. This is not a minor omission; all architecture conclusions depend on it.

The RadImageNet comparison is weaker than the abstract suggests. The authors admit the PyTorch weights perform differently and could not be fully matched to the TensorFlow ones, so the null result may be an artifact of framework mismatch rather than a true reflection of pretraining quality. And the headline AUCs are single runs without confidence intervals; 'optimal' is defined by validation AUC on the same data used for selection.

If the split is fixed and the comparison is done consistently in one framework, this would be a genuinely useful empirical contribution for people doing transfer learning on small medical datasets. As it stands, the claims are unverified but not implausible. I'd send it to peer review with a strong request to prove the grouped split and add error bars. It's the kind of paper a reading group could use to discuss evaluation pitfalls more than to learn the best architecture.

Referee Report

3 major / 5 minor

Summary. The paper reports transfer-learning experiments for two medical image classification tasks: ACL tear detection on the MRNet MRI dataset and breast nodule malignancy detection on a breast ultrasound dataset. The authors compare ImageNet- and RadImageNet-initialized backbones (ResNet50, InceptionV3, DenseNet121) under different classifier architectures, unfreezing strategies, optimizers, and augmentation policies. Their main claims are: (1) a 1-D convolutional classifier with skip connections, a ResNet50 pre-trained backbone, and partial backbone unfreezing is optimal; (2) the best models achieve AUCs of 0.9969 (ACL) and 0.9641 (breast); and (3) RadImageNet pre-training does not provide superior downstream performance compared to ImageNet pre-training. The paper includes extensive grid-search results, Grad-CAM visualizations, and appendices on baseline experiments and implementation details.

Significance. If the empirical findings were fully supported, the paper would provide useful, actionable guidance for practitioners choosing transfer-learning architectures in small-data medical imaging, and it would directly challenge the prior claim by Mei et al. (2022) that RadImageNet pre-training improves downstream performance. The systematic comparison across classifier head, backbone, unfreezing, and pretraining source is a valuable design. However, the paper's central empirical claims are currently not adequately supported because of unresolved data-splitting, implementation-matching, and statistical-rigor issues. The contribution is therefore conditional on a substantial re-analysis; the questions asked are important, but the manuscript in its present form does not yet answer them convincingly.

major comments (3)

[Table 1 and experimental setup] The MRNet dataset is described as containing 1021 knee MRI exams, but the input is listed as '.png MRI image' and the experimental task is per-image classification. No passage states that all slices from the same exam are kept in the same train/validation/test partition. If slices are split at image level, the same knee can appear in both training and test, allowing the model to memorize patient anatomy and inflating all reported AUCs. This is load-bearing: every headline result—ACL AUC 0.9969, breast AUC 0.9641, the ResNet50/ConvSkip/partial-unfreezing recommendation, and the RadImageNet comparison—depends on how the data were partitioned. The authors must either report a grouped (exam-level) split or re-run the experiments with such a split and report both per-image and per-exam metrics. Relatedly, Table 1 labels the ACL task output as 'meniscus tear' rather than ACL tear, which is an
[Appendix D.2] The comparison between RadImageNet and ImageNet pre-training is weakened by the acknowledged TensorFlow/PyTorch weight discrepancy. The authors state that TensorFlow implementations outperformed PyTorch for overlapping architectures and that PyTorch RadImageNet InceptionV3 was 'highly unstable.' If the final best-model experiments run in PyTorch, the observed lack of a RadImageNet advantage may reflect weight-conversion or framework artifacts rather than a genuine property of pretraining data. The manuscript needs either (a) a matched implementation with verified weight equivalence across frameworks, or (b) a sensitivity analysis showing the RadImageNet-versus-ImageNet conclusion is robust to the framework used. Without this, the null RadImageNet claim is not established.
[Appendix A and Appendix F] Table 9 reports single test AUCs with no confidence intervals, no repeated runs, and no measure of across-seed or across-split variance. The 'optimal' architecture is selected by validation AUC over a large grid, so the reported test performance is conditional on the validation fold; an unbiased estimate would require nested cross-validation or repeated resampling. The paper's claim to perform 'statistical analysis' comparing pretraining sources is not backed by the presented evidence: box plots of validation AUCs are descriptive, not inferential. At minimum, bootstrapped confidence intervals or repeated-split results are needed to support the optimality and comparison claims.

minor comments (5)

[Table 1] The output label for the ACL task is listed as 'meniscus tear' instead of 'ACL tear'; this contradicts the task description and should be corrected.
[Appendix E] 'RadIamgeNet' is a typo; should be 'RadImageNet'.
[Appendix F.2 captions] 'yieldes' should be 'yields' in the captions of Figures 28 and 32.
[Experimental reproducibility] The manuscript does not state the number of training epochs, the exact optimizer hyperparameters, the random seed policy, or whether code and model checkpoints will be released. Adding these details would substantially improve reproducibility.
[Reference [24]] The reference list entry for Sun et al. is incomplete; it lacks the publication venue and full bibliographic information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's conclusions are empirical benchmark results, not derivations; reported limitations are statistical validity issues, not circular reductions.

full rationale

The paper is an empirical study of transfer-learning architectures for two medical imaging tasks. It does not derive any quantity from first principles; every headline result (AUCs, architecture ranking, RadImageNet-vs-ImageNet comparison) is an experimental measurement on fixed train/validation/test splits. No fitted parameter is renamed as a prediction: hyperparameters and architectures are selected by validation AUC and then evaluated on a held-out test set, which is standard model selection rather than a circular reduction. The comparison with RadImageNet uses external pretrained weights from Mei et al. (2022), not the authors' own prior work; Appendix D explicitly notes that the PyTorch RadImageNet weights could not be perfectly matched to the TensorFlow weights, a limitation that weakens the null RadImageNet result but does not make the conclusion equivalent to its input. No uniqueness theorem, ansatz, or known result is imported via self-citation. The remaining concerns (possible slice-level leakage in the MRNet split, Table 1 labeling the ACL output as 'meniscus tear', and validation-based model selection) are correctness and reporting issues, not circularity under the required standard of exhibiting a specific reduction by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's conclusions rest on standard transfer learning practice plus empirical choices: which backbone, filters, schedule, and unfreezing strategy to use, and on the reliability of the pretrained weights and the data split. The most fragile are the fidelity of the RadImageNet PyTorch weights and the absence of a documented leakage-free split.

free parameters (5)

Backbone architecture = ResNet50
Selected by grid search; the paper claims ResNet50 is superior, but this is a choice fitted to the validation set.
Number of convolutional filters = 16
Selected by grid search; Appendix F shows 16 filters give better average validation AUC.
Learning rate schedule = Cosine annealing
Selected by grid search; cosine annealing yielded better average validation AUC than alternatives.
Unfreezing strategy = Partial backbone unfreezing
Chosen empirically as the best of freeze-all, unfreeze-all, and partial unfreezing.
Data augmentation policy = Flips, rotations, affine, no color jitter/blur
Augmentation choices were made based on validation AUC experiments.

assumptions (4)

domain assumption Labels in MRNet and Breast Ultrasound datasets are correct
The models are trained and evaluated against these labels; any label noise directly affects AUC scores.
domain assumption Open-source PyTorch RadImageNet weights are faithful to the official TensorFlow weights
The comparison of RadImageNet vs ImageNet assumes the weights used are valid; Appendix D shows instability and performance gaps, so this axiom is questionable.
domain assumption Train/validation/test splits prevent patient or exam-level leakage
The paper processes .png slices from MRI exams but does not state that all slices of an exam stay in one split; if not, AUC is inflated.
domain assumption Validation AUC is an unbiased selector for architecture comparison
The best architecture is selected on validation AUC, but no nested cross-validation is reported, so the selected model's test performance is optimistically biased.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection." pith.science (2026). https://pith.science/paper/TRIKW7EN

@misc{pith2026250817567,
  author       = {Pith},
  title        = {Pith review of: Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRIKW7EN}},
  note         = {Machine review of arXiv:2508.17567}
}

read the original abstract

Modern computer vision models have proven to be highly useful for medical imaging classification and segmentation tasks, but the scarcity of medical imaging data often limits the efficacy of models trained from scratch. Transfer learning has emerged as a pivotal solution to this, enabling the fine-tuning of high-performance models on small data. Mei et al. (2022) found that pre-training CNNs on a large dataset of radiologist-labeled images (RadImageNet) enhanced model performance on downstream tasks compared to ImageNet pretraining. The present work extends Mei et al. (2022) by conducting a comprehensive investigation to determine optimal CNN architectures for breast lesion malignancy detection and ACL tear detection, as well as performing statistical analysis to compare the effect of RadImageNet and ImageNet pre-training on downstream model performance. Our findings suggest that 1-dimensional convolutional classifiers with skip connections, ResNet50 pre-trained backbones, and partial backbone unfreezing yields optimal downstream medical classification performance. Our best models achieve AUCs of 0.9969 for ACL tear detection and 0.9641 for breast nodule malignancy detection, competitive with the results reported by Mei et al. (2022) and surpassing other previous works. We do not find evidence confirming RadImageNet pre-training to provide superior downstream performance for ACL tear and breast lesion classification tasks.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

[1]

Introduction Computer vision models have the power to en- able enhanced detection accuracy, better patient triage, and high-quality automated segmentation for medi- cal imaging ([5], [14], [24], [20], [10]), but data col- lection challenges often necessitate model develop- ment with small datasets. Traditionally, small data would provide insufficient lear...

work page 2022
[2]

The original RadImageNet work primarily focused on TensorFlow implementation, and it is possible that the TensorFlow pre-training weights were similarly given more focus and training time compared with PyTorch. In addition to any architectural differences between the implementations, the experiments are fundamentally different in that TensorFlow relies on...

work page
[3]

Ideally, as a sanity check of our PyTorch implementation, we would have derived nearly equivalent performance between the two implementations

In particular, InceptionV3 was highly unstable with the PyTorch RadImageNet weights, which creates a stronger performance gap between the two implementations for RadImageNet pre-training. Ideally, as a sanity check of our PyTorch implementation, we would have derived nearly equivalent performance between the two implementations. However, we are confident ...

work page
[13]

A large part of the challenge lies in the fact that TensorFlow abstracts much more logic away from the programmer, so we had to do our best to deduce default and hidden behaviors

While we exhaustively read through the legacy TensorFlow 2.0 code [2] used by the RadImageNet authors, it was not possible to perfectly match every single detail of the TensorFlow implementation. A large part of the challenge lies in the fact that TensorFlow abstracts much more logic away from the programmer, so we had to do our best to deduce default and...

work page
[14]

Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection

by elucidating the best architectural choices for developing transfer-learned CNN models for two ra- diology tasks: anterior cruciate ligament (ACL) tear detection Table 1 and breast nodule malignancy de- tection Table 2. Task Details Dataset MRNet dataset [23] Description Includes 1021 ACL knee MRI exams per- formed at Stanford University Medical Center....

work page Pith review arXiv 2025
[18]

R: A Language and Environment for Statistical Computing

R Core Team. R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria, 2021

work page 2021
[19]

Robin, N

X. Robin, N. Turck, A. Hainard, N. Tiberti, F. Lisacek, J.-C. Sanchez, and M. M ¨uller. proc: an open-source package for r and s+ to analyze and compare roc curves. BMC Bioinformatics, 12:77, 2011

work page 2011
[20]

N. E. Rose and S. M. Gold. A comparison of accu- racy between clinical examination and magnetic reso- nance imaging in the diagnosis of meniscal and ante- rior cruciate ligament tears. Arthroscopy, 12(4):398– 405, 1996

work page 1996

Show all 15 references

[21]

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explana- tions from deep networks via gradient-based localiza- tion. arXiv preprint arXiv:1610.02391, 2019

2019 arXiv
[22]

A. Shah. Breast Ultrasound Images Dataset. https://www.kaggle. com/datasets/aryashah2k/ breast-ultrasound-images-dataset , 2021

2021
[23]

MRNet Competition

Stanford ML Group. MRNet Competition. https://stanfordmlgroup.github.io/ competitions/mrnet/, 2023

2023
[24]

Y . Sun, Y . Qu, D. Wang, Y . Li, L. Ye, J. Du, B. Xu, B. Li, X. Li, K. Zhang, et al. Deep learning model improves radiologists’ performance in detection and classification of breast lesions. 2021. * These authors contributed equally to this work

2021
[25]

Wilcoxon

F. Wilcoxon. Individual comparisons by ranking meth- ods. Biometrics Bulletin, 1(6):80–83, 1945. Accessed 5 June 2024

1945
[26]

Zhang, Y

Q. Zhang, Y . N. Wu, and S.-C. Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020

2020
[27]

Z. Zhao, L. Alzubaidi, J. Zhang, Y . Duan, and Y . Gu. A comparison review of transfer learning and self- supervised learning: Definitions, applications, advan- tages and limitations. Expert Systems with Applica- tions, 2023. 10 Appendix A. Full Best Model Results Table 9: Bes...

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Introduction Computer vision models have the power to en- able enhanced detection accuracy, better patient triage, and high-quality automated segmentation for medi- cal imaging ([5], [14], [24], [20], [10]), but data col- lection challenges often necessitate model develop- ment with small datasets. Traditionally, small data would provide insufficient lear...

work page 2022

[2] [2]

The original RadImageNet work primarily focused on TensorFlow implementation, and it is possible that the TensorFlow pre-training weights were similarly given more focus and training time compared with PyTorch. In addition to any architectural differences between the implementations, the experiments are fundamentally different in that TensorFlow relies on...

work page

[3] [3]

Ideally, as a sanity check of our PyTorch implementation, we would have derived nearly equivalent performance between the two implementations

In particular, InceptionV3 was highly unstable with the PyTorch RadImageNet weights, which creates a stronger performance gap between the two implementations for RadImageNet pre-training. Ideally, as a sanity check of our PyTorch implementation, we would have derived nearly equivalent performance between the two implementations. However, we are confident ...

work page

[4] [13]

A large part of the challenge lies in the fact that TensorFlow abstracts much more logic away from the programmer, so we had to do our best to deduce default and hidden behaviors

While we exhaustively read through the legacy TensorFlow 2.0 code [2] used by the RadImageNet authors, it was not possible to perfectly match every single detail of the TensorFlow implementation. A large part of the challenge lies in the fact that TensorFlow abstracts much more logic away from the programmer, so we had to do our best to deduce default and...

work page

[5] [14]

Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection

by elucidating the best architectural choices for developing transfer-learned CNN models for two ra- diology tasks: anterior cruciate ligament (ACL) tear detection Table 1 and breast nodule malignancy de- tection Table 2. Task Details Dataset MRNet dataset [23] Description Includes 1021 ACL knee MRI exams per- formed at Stanford University Medical Center....

work page Pith review arXiv 2025

[6] [18]

R: A Language and Environment for Statistical Computing

R Core Team. R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria, 2021

work page 2021

[7] [19]

Robin, N

X. Robin, N. Turck, A. Hainard, N. Tiberti, F. Lisacek, J.-C. Sanchez, and M. M ¨uller. proc: an open-source package for r and s+ to analyze and compare roc curves. BMC Bioinformatics, 12:77, 2011

work page 2011

[8] [20]

N. E. Rose and S. M. Gold. A comparison of accu- racy between clinical examination and magnetic reso- nance imaging in the diagnosis of meniscal and ante- rior cruciate ligament tears. Arthroscopy, 12(4):398– 405, 1996

work page 1996