{"id":"3a1153ab-e42f-4d5d-9fc2-4629f2c849e4","arxiv_id":"2508.17567","paper_version":2,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On two public medical imaging tasks, 1D convolutional classifiers with skip connections, ResNet50 backbones, and partial unfreezing gave the best validation AUC, while RadImageNet pretraining did not beat ImageNet.","lead":"This paper benchmarks convolutional transfer learning for two medical imaging tasks: ACL tear detection in knee MRIs and breast lesion malignancy classification. It reports top AUCs of 0.9969 and 0.9641 and finds no clear benefit from RadImageNet over ImageNet pretraining.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unstated exam-level train/test split in the MRNet experiments can inflate all reported AUCs and architecture rankings; the paper must demonstrate a grouped split or the central claims are unverified.","rationale":"The core claims all flow from the experimentally measured test AUCs. In slice-based medical imaging, the unit of randomization must be the exam/patient, not the image; otherwise the test set contains near-duplicates of training data. The paper's own Table 1 calls the data '1021 exams' while the input is a single '.png MRI image,' and no grouping statement appears anywhere in the provided text. This is a missing control that would change every conclusion if wrong. The reader flagged exactly this; my independent read agrees. Appendix D's admission about RadImageNet weight mismatch is a separate weakness that mainly undermines the null comparison; I would not let it overshadow the split issue. I therefore do not think the present text can support a final REJECT as a scientific conclusion from evidence; it can support a conditional stance until the split is demonstrated. I recommend CONDITIONAL: require a grouped (exam-level) split and a rerun of the best configurations; if the AUCs survive, the central empirical claims are substantially supported.","tokens_in":5060,"tokens_out":8402,"duration_ms":108609,"concrete_test":"Obtain or reconstruct the exact data partition. If the split was image-level, rerun the best ACL and breast configurations (same hyperparameters, same augmentation) using GroupShuffleSplit on exam/patient ID so all slices of an exam remain in one fold. If the grouped-split test AUCs stay close to 0.9969/0.9641, leakage did not produce the result; if AUC drops materially (e.g., ACL below ~0.95), the headline claims collapse. In addition, verify that the ACL task labels are ACL tears, not the 'meniscus tear' label given in Table 1.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Every headline number — ACL AUC 0.9969, breast AUC 0.9641, the ResNet50/ConvSkip/partial-unfreezing recommendation, and the RadImageNet-versus-ImageNet comparison — depends on how the MRNet images were partitioned. The paper describes the ACL task as 1021 exams but uses '.png MRI image' as input, and no passage in the provided text states that slices from one exam are kept in the same train/validation/test fold. If the split is at image level, the same knee exam appears in both training and test, so the model can memorize patient anatomy and all reported AUCs are inflated. This is not merely a missing detail: Table 1 even labels the ACL output as 'meniscus tear,' an internal inconsistency that casts doubt on dataset bookkeeping. Separately, Appendix D admits the PyTorch RadImageNet weights could not be fully matched to the TensorFlow weights, weakening the null RadImageNet result. But the split issue is decisive because it affects even the ImageNet-based architectural conclusions, not just the comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports transfer-learning experiments for two medical image classification tasks: ACL tear detection on the MRNet MRI dataset and breast nodule malignancy detection on a breast ultrasound dataset. The authors compare ImageNet- and RadImageNet-initialized backbones (ResNet50, InceptionV3, DenseNet121) under different classifier architectures, unfreezing strategies, optimizers, and augmentation policies. Their main claims are: (1) a 1-D convolutional classifier with skip connections, a ResNet50 pre-trained backbone, and partial backbone unfreezing is optimal; (2) the best models achieve AUCs of 0.9969 (ACL) and 0.9641 (breast); and (3) RadImageNet pre-training does not provide superior downstream performance compared to ImageNet pre-training. The paper includes extensive grid-search results, Grad-CAM visualizations, and appendices on baseline experiments and implementation details.","tokens_in":5320,"tokens_out":3725,"duration_ms":47466,"significance":"If the empirical findings were fully supported, the paper would provide useful, actionable guidance for practitioners choosing transfer-learning architectures in small-data medical imaging, and it would directly challenge the prior claim by Mei et al. (2022) that RadImageNet pre-training improves downstream performance. The systematic comparison across classifier head, backbone, unfreezing, and pretraining source is a valuable design. However, the paper's central empirical claims are currently not adequately supported because of unresolved data-splitting, implementation-matching, and statistical-rigor issues. The contribution is therefore conditional on a substantial re-analysis; the questions asked are important, but the manuscript in its present form does not yet answer them convincingly.","major_comments":[{"comment":"The MRNet dataset is described as containing 1021 knee MRI exams, but the input is listed as '.png MRI image' and the experimental task is per-image classification. No passage states that all slices from the same exam are kept in the same train/validation/test partition. If slices are split at image level, the same knee can appear in both training and test, allowing the model to memorize patient anatomy and inflating all reported AUCs. This is load-bearing: every headline result—ACL AUC 0.9969, breast AUC 0.9641, the ResNet50/ConvSkip/partial-unfreezing recommendation, and the RadImageNet comparison—depends on how the data were partitioned. The authors must either report a grouped (exam-level) split or re-run the experiments with such a split and report both per-image and per-exam metrics. Relatedly, Table 1 labels the ACL task output as 'meniscus tear' rather than ACL tear, which is an","section":"Table 1 and experimental setup"},{"comment":"The comparison between RadImageNet and ImageNet pre-training is weakened by the acknowledged TensorFlow/PyTorch weight discrepancy. The authors state that TensorFlow implementations outperformed PyTorch for overlapping architectures and that PyTorch RadImageNet InceptionV3 was 'highly unstable.' If the final best-model experiments run in PyTorch, the observed lack of a RadImageNet advantage may reflect weight-conversion or framework artifacts rather than a genuine property of pretraining data. The manuscript needs either (a) a matched implementation with verified weight equivalence across frameworks, or (b) a sensitivity analysis showing the RadImageNet-versus-ImageNet conclusion is robust to the framework used. Without this, the null RadImageNet claim is not established.","section":"Appendix D.2"},{"comment":"Table 9 reports single test AUCs with no confidence intervals, no repeated runs, and no measure of across-seed or across-split variance. The 'optimal' architecture is selected by validation AUC over a large grid, so the reported test performance is conditional on the validation fold; an unbiased estimate would require nested cross-validation or repeated resampling. The paper's claim to perform 'statistical analysis' comparing pretraining sources is not backed by the presented evidence: box plots of validation AUCs are descriptive, not inferential. At minimum, bootstrapped confidence intervals or repeated-split results are needed to support the optimality and comparison claims.","section":"Appendix A and Appendix F"}],"minor_comments":[{"comment":"The output label for the ACL task is listed as 'meniscus tear' instead of 'ACL tear'; this contradicts the task description and should be corrected.","section":"Table 1"},{"comment":"'RadIamgeNet' is a typo; should be 'RadImageNet'.","section":"Appendix E"},{"comment":"'yieldes' should be 'yields' in the captions of Figures 28 and 32.","section":"Appendix F.2 captions"},{"comment":"The manuscript does not state the number of training epochs, the exact optimizer hyperparameters, the random seed policy, or whether code and model checkpoints will be released. Adding these details would substantially improve reproducibility.","section":"Experimental reproducibility"},{"comment":"The reference list entry for Sun et al. is incomplete; it lacks the publication venue and full bibliographic information.","section":"Reference [24]"}],"recommendation":"major_revision","confidential_remarks":"The data-splitting issue is potentially decisive. If the authors cannot demonstrate a grouped split, the paper may not be salvageable because all headline AUCs and architectural rankings would be untrustworthy. However, since the fix is within the authors' control (re-running with an exam-level split), a major revision is more appropriate than a rejection at this stage. The RadImageNet weight-matching issue is also significant and should be resolved before any claim about pretraining source superiority is made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my honest read. The paper is a careful empirical grid search, not a new method. It does useful things: it compares three classifier heads, three backbones, several unfreezing schemes, and learning-rate schedules on two medical imaging datasets, and it shows box plots of validation AUC/F1/accuracy rather than cherry-picking a single run. The authors also deserve credit for documenting the TensorFlow/PyTorch weight discrepancy in Appendix D; that transparency is rare.\n\nThe problems are the ones you flagged. The split is the load-bearing one. MRNet is 1021 exams but the input is per-slice .png images, and the paper never says that slices from the same exam are kept together. If they aren't, the model memorizes patient anatomy and every number in the paper is inflated. The Table 1 typo—'output label for meniscus tear' in the ACL task—makes the bookkeeping look sloppy, and there is no methods section in the version I saw to check. This is not a minor omission; all architecture conclusions depend on it.\n\nThe RadImageNet comparison is weaker than the abstract suggests. The authors admit the PyTorch weights perform differently and could not be fully matched to the TensorFlow ones, so the null result may be an artifact of framework mismatch rather than a true reflection of pretraining quality. And the headline AUCs are single runs without confidence intervals; 'optimal' is defined by validation AUC on the same data used for selection.\n\nIf the split is fixed and the comparison is done consistently in one framework, this would be a genuinely useful empirical contribution for people doing transfer learning on small medical datasets. As it stands, the claims are unverified but not implausible. I'd send it to peer review with a strong request to prove the grouped split and add error bars. It's the kind of paper a reading group could use to discuss evaluation pitfalls more than to learn the best architecture.","headline":"Solid empirical grid search, but the headline AUCs rest on an unstated data-split assumption that could invalidate the rankings.","tokens_in":5712,"tokens_out":3457,"would_cite":false,"duration_ms":40985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the best transfer-learning recipe for two small medical-imaging datasets is an ImageNet-pretrained ResNet50 backbone, partially unfrozen, with a 1D convolutional skip-connection head—and that RadImageNet pretraining gi","keywords":["transfer learning","convolutional neural networks","medical image classification","ACL tear detection","breast lesion malignancy","ImageNet pretraining","RadImageNet","ResNet50"],"falsifier":"Re-run the best ACL model with a strict exam-level split, placing all slices from one knee exam in only one partition, and compare the test AUC to 0.9969; a large drop would show the original number depended on slice-level leakage. A similar reconstruction for the breast task, grouping images by patient if identifiers are available, would test that result as well.","tokens_in":4981,"feed_emoji":"🩻","tokens_out":8204,"duration_ms":90175,"temperature":0.7,"pith_summary":"This paper asks what a practitioner should actually do when fine-tuning a convolutional network on a small medical imaging dataset, and answers with a concrete recipe tested on two tasks: ACL tear detection from knee MRI and breast nodule malignancy detection from ultrasound. It finds that an ImageNet-pretrained ResNet50 backbone, partially unfrozen, with a lightweight 1D convolutional classifier that has skip connections, produces the best validation AUCs in a broad grid search—0.9969 for ACL tears and 0.9641 for breast malignancy on the test sets. It also finds no evidence that pretraining on RadImageNet, a large radiologist-labeled corpus, beats ImageNet pretraining on either task, directly challenging a prior influential claim. If the result holds, it simplifies transfer-learning choices for medical imaging: generic ImageNet features plus a small residual conv head are a strong default, and domain-specific pretraining should be justified per task rather than assumed.","feed_headline":"ImageNet pretraining matches RadImageNet for two medical tasks","feed_subtitle":"Best recipe found: ResNet50 backbone, partial unfreezing, 1D conv head with skip connections; AUCs 0.997 and 0.964.","key_machinery":"The mechanism that drives the result is the fine-tuning setup: take a ResNet50 backbone pre-trained on ImageNet, keep most layers frozen, unfreeze the later blocks, and attach a small classifier made of 1D convolutions with a skip connection. The skip connection acts like a residual block, letting fine-tuning adjust high-level features without destroying the pre-trained representation. Cosine annealing learning-rate decay and light augmentation (flips, rotations, affine transforms) are the supporting controls that stabilize training, and the grid search across backbones, classifier heads, and freezing strategies is what lets the paper attribute gains to these specific choices.","core_discovery":"The core claim is that optimal downstream medical classification performance comes from ImageNet-pretrained ResNet50 backbones combined with a 1D convolutional classifier with skip connections, trained with partial backbone unfreezing and cosine annealing learning rate decay. Across a grid search over backbones, classifier heads, and unfreezing strategies, this configuration consistently ranked first on validation AUC, F1, and accuracy for both ACL tear detection and breast nodule malignancy detection. The paper further claims that RadImageNet pretraining did not confer a downstream advantage: ImageNet-initialized models showed better convergence and higher validation AUC in both tasks. Best","pith_inferences":["An editorial test this paper does not run: repeat the best ACL model with all slices from the same knee exam confined to one train/validation/test split; if the 0.9969 AUC holds, the recipe is robust, and if it drops, the headline numbers are partly inflated by slice-level leakage.","The lack of a RadImageNet advantage may stem from these tasks relying on generic edge and texture features that ImageNet already captures; a natural extension is to compare both pretraining corpora on modalities with stronger distribution shift, such as CT or X-ray.","Because no confidence intervals are reported for the best-model AUCs, bootstrap or repeated-seed intervals would clarify whether the architecture differences are real or within run-to-run noise."],"forward_implications":["If the recipe is followed, fine-tuning on small medical datasets can default to an ImageNet-pretrained ResNet50 with a skip-connection conv head rather than heavier or more elaborate architectures.","Domain-specific pretraining corpora should not be assumed superior; task-specific comparison is needed before adopting them.","Partial unfreezing of a ResNet50 backbone beat both fully frozen and fully unfrozen training, making it a strong default schedule.","The grid search rankings by validation AUC mostly matched rankings by F1 and accuracy, so AUC appears to be a reliable selection criterion for these tasks.","The reported test AUCs, 0.9969 for ACL tears and 0.9641 for breast malignancy, are competitive with or better than previously published pipelines on the same tasks."],"supporting_citations":[{"why":"Defines RadImageNet pretraining and the claim of downstream superiority that this paper tests and does not replicate.","marker":"[14]"},{"why":"Supplies the knee MRI dataset used for ACL tear detection.","marker":"[23]"},{"why":"Supplies the breast ultrasound images used for malignancy classification.","marker":"[22]"},{"why":"Provides the prior breast-lesion deep learning result the paper compares against and surpasses.","marker":"[24]"},{"why":"Provides the Grad-CAM method used for qualitative assessment of model attention.","marker":"[21]"},{"why":"Provides the ImageNet natural-image pretraining dataset used as the comparison baseline.","marker":"[7]"},{"why":"Supplies the RadImageNet pretrained weights and baseline code the authors adapt for their experiments.","marker":"[4]"},{"why":"Provides the method used to compare ROC curves in the statistical analysis of pretraining effects.","marker":"[19]"}],"fun_headline_variants":["ResNet50, partial unfreezing, 1D conv head win medical imaging tests","ImageNet beats RadImageNet for ACL tear and breast lesion detection","Best CNN recipe: ResNet50, 1D conv classifier, partial backbone unfreezing","RadImageNet offers no advantage over ImageNet in two medical tasks","High AUCs in ACL and breast tasks with ResNet50 and 1D conv head"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper treats each MRI slice as an independent sample without stating that every slice from the same knee exam is assigned to the same train/validation/test split; if slices from one exam cross splits, the reported AUCs are inflated by patient-level leakage.","fun_headline_variants_meta":{"raw":{"variants":["ResNet50, partial unfreezing, 1D conv head win medical imaging tests","ImageNet beats RadImageNet for ACL tear and breast lesion detection","Best CNN recipe: ResNet50, 1D conv classifier, partial backbone unfreezing","RadImageNet offers no advantage over ImageNet in two medical tasks","High AUCs in ACL and breast tasks with ResNet50 and 1D conv head"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1523,"prompt_tokens":766,"completion_tokens":757,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":510,"tokens_out":757,"duration_ms":8217,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:51:26.434333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the best ACL model with a strict exam-level split, placing all slices from one knee exam in only one partition, and compare the test AUC to 0.9969; a large drop would show the original number depended on slice-level leakage. A similar reconstruction for the breast task, grouping images by patient if identifiers are available, would test that result as well.","supporting_citations":[{"cited_title":"Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection","cited_arxiv_id":"2508.17567","evidence_quote":"Defines RadImageNet pretraining and the claim of downstream superiority that this paper tests and does not replicate."},{"cited_title":"MRNet Competition","cited_arxiv_id":null,"evidence_quote":"Supplies the knee MRI dataset used for ACL tear detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the breast ultrasound images used for malignancy classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior breast-lesion deep learning result the paper compares against and surpasses."},{"cited_title":"Robin, N","cited_arxiv_id":null,"evidence_quote":"Provides the method used to compare ROC curves in the statistical analysis of pretraining effects."}],"review_version":1}