{"id":"7b933c2d-1bbc-46ea-ad57-e30f28038fdf","arxiv_id":"2505.19779","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning recent natural-domain foundation models, especially AIMv2, improves medical image classification accuracy across mammography, skin lesion, retinopathy, and chest X-ray benchmarks.","lead":"This paper fine-tunes seven recent natural-image foundation models, including AIMv2, DINOv2 and SAM2, on four medical imaging datasets. It reports that the newest models transfer best, with AIMv2 leading across all four tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Tables 2 and 3 contradict the central claim that AIMv2 achieved the highest performance in all frozen/unfrozen configurations across all four datasets.","rationale":"I read the paper as a benchmarking study whose central assertion is that AIMv2, followed by DINOv2 and SAM2, outperforms older foundation models across four medical datasets, demonstrating that progress in natural-domain training transfers to medical image classification. For that assertion to hold, the reported tables must support the claimed ranking. They do not. The contradiction in Tables 2 and 3 is a factual issue that does not depend on any external assumptions. The leakage concern raised by the reader is legitimate—CheXpert and CBIS-DDSM have multiple images per patient, and an image-level 80/10/10 split can leak patient information—but fixing the split would not resolve the internal inconsistency; even the current tables fail to substantiate the headline claim. The paper does have genuine strengths: it evaluates seven recent models with multiple backbones, reports frozen and unfrozen settings with two head types, and makes code public. These are useful. However, the conclusion overstates what the data show. I therefore retain the CONDITIONAL verdict, but the required condition should be a correction of the ranking claim and the addition of uncertainty quantification, not only patient-level splits.","tokens_in":8540,"tokens_out":8916,"duration_ms":81605,"concrete_test":"Recompute the per-dataset, per-configuration maxima from Tables 1–4 and list which model reaches each maximum. If APTOS frozen-linear resolves to DINOv2 ViT-L (0.889) and ISIC frozen-attention resolves to MAE ViT-H (0.802), the claim in §3.2 is contradicted; the authors should then either revise the claim to name the actual per-configuration leaders or provide repeated-seed results with confidence intervals showing whether AIMv2's remaining advantages (e.g., 0.001–0.002 on APTOS unfrozen-linear) are statistically distinguishable from the runner-up.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that 'AIMv2 achieved the highest performance across both frozen and unfrozen configurations in all four datasets,' but the reported tables contradict this. In Table 2 (APTOS), the frozen-linear maximum is DINOv2 ViT-L (0.889), while AIMv2 ViT-L scores 0.888 and AIMv2 ViT-H scores 0.885; in the unfrozen-attention column CLIP ViT-B ties AIMv2 ViT-L at 0.907. In Table 3 (ISIC2019), the frozen-attention maximum is MAE ViT-H (0.802), exceeding AIMv2 ViT-H (0.795). Thus the central ranking claim is not merely unproven; it is false as written for at least two configurations in two datasets. Furthermore, many of the margins used to rank models are 0.001–0.002 (e.g., APTOS unfrozen linear AIMv2 0.915 vs DINOv2 0.913), and the paper reports no error bars or statistical tests, so the 'highest' designation is not statistically meaningful. The absence of patient-level splits (CheXpert and CBIS-DDSM contain multiple images per patient) is a separate validity threat, but the internal contradiction is the more immediate problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates seven recent vision foundation models (CLIP, DINOv2, MAE, VMamba, CoCa, SAM2, AIMv2) on four medical image classification datasets (CBIS-DDSM, APTOS2019, ISIC2019, CHEXPERT), under four configurations: frozen or unfrozen backbones with linear or multi-layer attention heads. The main empirical claim, stated in Section 3.2 and the Conclusion, is that AIMv2 achieves the highest performance across both frozen and unfrozen configurations in all four datasets, with DINOv2 and SAM2 following closely, and that recent advances in natural-domain training transfer positively to medical imaging. The paper also reports configuration-dependent findings on linear versus attention heads and on model-size trends, and it makes the code publicly available.","tokens_in":8871,"tokens_out":4171,"duration_ms":41185,"significance":"If substantiated, the paper would provide practically useful guidance for selecting foundation models for medical classification transfer, especially the recommendation of AIMv2, DINOv2, and SAM2 over older models such as CLIP and MAE. The study covers a broad and current set of models, applies four standard medical benchmarks, and releases code, which are strengths. However, the central ranking claim is contradicted by the paper's own tables in at least two configurations, and the evaluation lacks error bars, multiple-seed statistics, or significance tests. The absence of patient-level splitting for datasets with multiple images per patient is a further validity threat. The underlying benchmark is reproducible in principle, but the headline conclusions are not yet supported as stated.","major_comments":[{"comment":"The claim that 'AIMv2 achieved the highest performance across both frozen and unfrozen configurations in all four datasets' is contradicted by the reported numbers. In Table 2 (APTOS2019), the frozen-linear maximum is DINOv2 ViT-L at 0.889, while AIMv2 ViT-L is 0.888; in the unfrozen-attention column, CLIP ViT-B ties AIMv2 ViT-L at 0.907. In Table 3 (ISIC2019), the frozen-attention maximum is MAE ViT-H at 0.802, exceeding AIMv2 ViT-H at 0.795. The text in Section 3.2 and the Conclusion should be corrected to describe the actual per-configuration maxima or should explicitly qualify the ranking with these exceptions.","section":"Section 3.2, Tables 1-4"},{"comment":"The evaluation provides no error bars, no multiple-seed statistics, and no significance tests, yet the Conclusion states that the models 'significantly enhance' outcomes and the ranking labels models as 'highest.' Several margins used for ranking are 0.001-0.003, for example APTOS2019 unfrozen-linear AIMv2 0.915 versus DINOv2 0.913, CHEXPERT unfrozen-linear AIMv2 0.819 versus DINOv2 0.814, and CBIS-DDSM unfrozen-linear AIMv2 0.968 versus DINOv2 0.966. These differences are within plausible run-to-run variation, so the ranking claims are not statistically supported. Please report multiple runs with confidence intervals or paired significance tests, and soften the language accordingly.","section":"Section 3.2 and Conclusion"},{"comment":"The data split is described as a random 80/10/10 division of images, without patient-level separation. CheXpert and CBIS-DDSM contain multiple images per patient, so images from the same patient can appear in both training and test sets, which can inflate AUC and kappa values through leakage. This could materially change the reported rankings, especially at the small margins observed. The authors should use patient-exclusive stratified splits, or provide a clear justification for why such leakage is negligible for the reported comparisons.","section":"Section 3.1, data split"}],"minor_comments":[{"comment":"The text states that ISIC2019 includes 'nine diagnostic categories' and then lists 'benign keratosis' and 'seborrheic keratosis' as separate entries. In the standard ISIC 2019 challenge, there are eight lesion categories, with benign keratosis (BKL) encompassing seborrheic keratosis. Please correct the description and the class count.","section":"Section 2.2, ISIC2019"},{"comment":"The phrase 'we used a pre-trained transformer [12]' is misleading: reference [12] is a paper on transfer learning for medical images, not a transformer architecture definition. Please cite the appropriate ViT or transformer reference, or rephrase to say a multi-layer Transformer head was used.","section":"Section 3.1, attention head"},{"comment":"The metric is called 'Quadratic Cohen Kappa'; the standard term is 'quadratic weighted Cohen's kappa.' Please use the correct name.","section":"Section 2.2, APTOS2019"},{"comment":"For grayscale datasets, the authors remove the second and third channel weights from the patch embedding layer. This is a reasonable design choice, but the paper does not compare it with alternatives such as averaging the three channels or replicating the gray image to three channels. A brief sensitivity note would strengthen the methodology.","section":"Section 3.1, grayscale adaptation"},{"comment":"The text in Section 3.4 refers to 'fig. 1' for the model-size comparison, but the figure and its caption are not included in the submitted text. Please ensure the figure is present and that the axes and compared configurations are clearly described.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This is a useful empirical benchmark with public code, but the headline ranking claim is factually contradicted by the paper's own tables in specific configurations. The lack of statistical grounding and patient-level splitting are additional concerns. I would support publication after the central claims are corrected and the evaluation protocol is strengthened, but not in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a legitimate extension of Huix et al.'s medical transfer benchmark: the authors add AIMv2, SAM2, VMamba, and CoCa, evaluate on four public datasets, and release code. Second, the paper's central claim—that AIMv2 achieved the highest performance across both frozen and unfrozen configurations in all four datasets—is contradicted by its own tables. In APTOS (Table 2) the frozen-linear maximum is DINOv2 ViT-L at 0.889, not AIMv2's 0.888; in ISIC2019 (Table 3) the frozen-attention maximum is MAE ViT-H at 0.802, above AIMv2 ViT-H's 0.795. So the stress-test note is right, and the abstract/conclusion overstate the results.\n\nWhat is genuinely useful: the new model rankings, the consistent finding that newer natural-domain models (DINOv2, AIMv2, SAM2) generally beat older CLIP/MAE, and the observation that fine-tuning helps. The paper follows the protocol of [13] transparently, and the public code improves reproducibility. That is real value for practitioners choosing a backbone.\n\nThe soft spots are proportional but real. The internal contradiction is load-bearing, since the abstract and conclusion repeat the 'highest in all configurations' phrasing. Many of the margins separating top models are 0.001–0.002, with no error bars, multiple seeds, or significance tests, so 'significantly enhance' is not supported. The random 80/10/10 split with no patient-level separation risks leakage in CheXpert and CBIS-DDSM, which contain multiple images per patient; a strict patient-exclusive split could change some rankings.\n\nNone of this makes the paper worthless. The trend claim—AIMv2/DINOv2/SAM2 are the best choices in most configurations—survives, but the absolute 'highest everywhere' phrasing needs to be corrected.\n\nMy recommendation: send it to peer review, but require a revision that fixes the overclaim, adds at least multiple-seed statistics or error bars, and clarifies the data split. With those changes it is a citable benchmark.","headline":"Useful benchmark extension, but the central claim that AIMv2 wins every configuration is contradicted by the paper's own tables.","tokens_in":9327,"tokens_out":2619,"would_cite":false,"duration_ms":24169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning recent natural-domain foundation models—AIMv2 above all—improves medical image classification across mammography, skin lesions, diabetic retinopathy, and chest X-rays.","keywords":["foundation models","medical image classification","transfer learning","fine-tuning","AIMv2","DINOv2","SAM2","chest X-ray"],"falsifier":"Recompute all four benchmarks under a patient-exclusive split, where every image of a given patient is assigned to one fold. If AIMv2's margin over DINOv2 and SAM2 shrinks or the order changes when this split is enforced, the paper's central ranking would be shown to depend on data leakage rather than on model quality.","tokens_in":8334,"feed_emoji":"🩻","tokens_out":6259,"duration_ms":56288,"temperature":0.7,"pith_summary":"This paper asks whether the recent wave of natural-domain vision foundation models, trained on web images, captions, or segmentation data, can be transferred to medical image classification. It fine-tunes seven models on four public medical datasets spanning mammography, skin lesions, diabetic retinopathy, and chest X-rays, comparing frozen and fully fine-tuned backbones with two classification heads. The central claim is that the newest models, AIMv2, DINOv2, and SAM2, outperform earlier ones such as CLIP and MAE across all four tasks, evidence that progress in natural image training carries over to medicine. A sympathetic reading of the results supports the recommendation that practitioners fine-tune these newer encoders rather than rely on older transfer-learning backbones.","feed_headline":"AIMv2 tops medical image benchmarks across four datasets","feed_subtitle":"Fine-tuned natural-domain backbones transfer well; DINOv2 and SAM2 follow close behind.","key_machinery":"The carrier of the argument is a controlled transfer-learning protocol: each encoder is kept as a feature extractor, a simple linear head or a multi-layer attention head is placed on top, and the backbone is either frozen or fine-tuned end to end. For grayscale inputs, the authors modify the patch-embedding layer by keeping only the first RGB channel's weights, allowing models pre-trained on color images to accept single-channel X-rays and mammograms without architectural change. The comparison across model families and backbone sizes, all evaluated at 224 by 224 resolution on the same four datasets with the same optimizer and learning-rate search, is what lets the authors attribute performance differences to the pre-training method. The multi-layer attention head, applied to patch embeddings rather than the classification token, is the component that most changes behavior between frozen and unfrozen modes.","core_discovery":"On the paper's own terms, the discovery is a clean empirical ranking: AIMv2 achieves the highest ROC-AUC, quadratic Cohen kappa, or recall in all four datasets under both frozen and unfrozen configurations, with DINOv2 and SAM2 close behind. The paper interprets this as evidence that each generation of natural-domain pre-training, from CLIP's contrastive language-image training to MAE's masked reconstruction to DINOv2's self-distillation and AIMv2's multimodal autoregression, improves the features that medical classifiers can reuse. It also reports that unfreezing the backbone consistently improves results and that a multi-layer attention head is better than a linear head when features are frozen. The result is intended to show that domain-specific medical pre-training is not required for strong classification performance and that advances in the natural domain transfer directly.","pith_inferences":["The paper's random 80/10/10 split may leak patient information: CheXpert and CBIS-DDSM contain multiple images per patient, and without patient-level separation the reported AUC and kappa could be inflated; a patient-exclusive split is the natural stress test of the ranking.","The claim that natural-domain progress transfers to medicine is inferred from a historical correlation across model families; the paper does not isolate which pre-training objective, contrastive, masked, autoregressive, or segmentation, drives the improvement.","The single-channel patch-embedding trick could be tested beyond classification, for example in medical segmentation or detection, where SAM2's own dense features may transfer even better than the classification heads used here.","Because only one or two backbone sizes per family are tested, the ranking may partly reflect model capacity and training budget rather than pre-training method; matching parameter counts across families would separate those factors."],"forward_implications":["If the ranking holds, practitioners should prefer AIMv2, DINOv2, or SAM2 as backbones for medical classification rather than older CLIP or MAE encoders, even on small labeled datasets.","Fine-tuning the backbone should be considered a default choice: the paper finds it consistently improves over frozen-feature classification, often making linear heads competitive with attention heads.","With frozen backbones, the multi-layer attention head is the better classifier; the paper reports it outperforms linear heads on all four datasets in this configuration.","Model-size choice depends on dataset size: base and large backbones win on smaller datasets, while the largest backbones win on the larger ISIC2019 and CHEXPERT collections.","The gains of AIMv2, DINOv2, and SAM2 come with lower training time than CoCa, making the best-performing models also the more efficient ones in these experiments."],"supporting_citations":[{"why":"The AIMv2 model, the paper's top performer across all four datasets; its multimodal autoregressive pre-training is the main innovation under test.","marker":"[9]"},{"why":"DINOv2, the self-supervised baseline that finishes second overall and supplies the strongest frozen-feature representations among non-AIMv2 models.","marker":"[6]"},{"why":"SAM2, the segmentation foundation model whose Hiera backbones are evaluated and reported as the third-best performer.","marker":"[8]"},{"why":"The prior study this paper follows up on, providing the configuration template of frozen versus unfrozen backbones and linear versus attention heads.","marker":"[13]"},{"why":"CBIS-DDSM mammography dataset, one of the four evaluation benchmarks and the source of the AUC numbers in Table 1.","marker":"[22]"},{"why":"APTOS2019 retinal image dataset, one of the four benchmarks and the source of the quadratic Cohen kappa metric.","marker":"[23]"},{"why":"ISIC2019 skin lesion dataset, one of the four benchmarks and the source of the recall metric.","marker":"[26]"},{"why":"CheXpert chest radiograph dataset, the largest benchmark in the study and a key source for the claim that large backbones win on large datasets.","marker":"[27]"},{"why":"CLIP, the oldest contrastive image-text baseline that the newer models are compared against.","marker":"[4]"},{"why":"MAE, the masked-autoencoder baseline family with ViT-B, ViT-L, and ViT-H backbones used across all four tasks.","marker":"[5]"}],"fun_headline_variants":["Natural foundation models excel in medical image classification","Fine-tuned AI models top medical benchmarks across four datasets","AIMv2 leads medical imaging after natural-domain fine-tuning","DINOv2, AIMv2, SAM2 shine in medical image tasks","Natural-domain models shine in medical image classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that a random 80/10/10 split of image files, rather than patients, gives unbiased test scores; with CheXpert and CBIS-DDSM containing repeated images of the same patient, leakage could inflate the reported metrics and alter the ranking.","fun_headline_variants_meta":{"raw":{"variants":["Natural foundation models excel in medical image classification","Fine-tuned AI models top medical benchmarks across four datasets","AIMv2 leads medical imaging after natural-domain fine-tuning","DINOv2, AIMv2, SAM2 shine in medical image tasks","Natural-domain models shine in medical image classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3805,"prompt_tokens":930,"completion_tokens":2875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2794}},"tokens_in":546,"tokens_out":2875,"duration_ms":20095,"temperature":1.0,"reasoning_tokens":2794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:05:17.123124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute all four benchmarks under a patient-exclusive split, where every image of a given patient is assigned to one fold. If AIMv2's margin over DINOv2 and SAM2 shrinks or the order changes when this split is enforced, the paper's central ranking would be shown to depend on data leakage rather than on model quality.","supporting_citations":[{"cited_title":"By fine-tuning these models on diverse datasets such as CBIS-DDSM, ISIC2019, APTOS2019, and CHEXPERT, we have demonstrated their capabil- ity to improve classification performance","cited_arxiv_id":null,"evidence_quote":"SAM2, the segmentation foundation model whose Hiera backbones are evaluated and reported as the third-best performer."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understand- ing,","cited_arxiv_id":null,"evidence_quote":"The prior study this paper follows up on, providing the configuration template of frozen versus unfrozen backbones and linear versus attention heads."},{"cited_title":"Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation,","cited_arxiv_id":null,"evidence_quote":"CBIS-DDSM mammography dataset, one of the four evaluation benchmarks and the source of the AUC numbers in Table 1."},{"cited_title":"Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models","cited_arxiv_id":"2505.19779","evidence_quote":"CLIP, the oldest contrastive image-text baseline that the newer models are compared against."}],"review_version":1}