{"id":"90d93d18-b576-4964-95cb-0c011407b391","arxiv_id":"2412.20172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new metric combining NCA-based feature separability with a conv2/conv1 gradient ratio ranks medical image source models better than six existing metrics on most targets, though statistical significance is limited.","lead":"The paper introduces a transferability score for medical image classification that combines how well target features separate in a pretrained model with the size of gradients in early layers during a single backward pass. It also releases a ground-truth transfer benchmark covering 15 source datasets and 9 architectures across 11 MedMNIST targets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-domain claim rests on a post hoc sign reversal of S_FU (Eq. 8 vs Eq. 5); without it, Ours is negatively correlated on most targets, so 'outperforms in both settings' is unsupported.","rationale":"The most load-bearing concern is the post hoc reversal of the gradient term's sign in the cross-domain experiment. The paper's central selling point is that S = S_LP * S_FU balances feature suitability with adaptability, with the hypothesis that larger early-layer gradient ratios imply better transferability. In Table 3, however, Ours only ranks first after applying Eq. 8, which flips the normalization of S_FU, and after applying a 1-S transform the same transform is selectively applied to PARC, NCTI, LEEP, and NLEEP. The authors admit in the text that this 'contradicts intuition' and that cross-domain transfer may 'operate differently.' That admission undercuts the proposed mechanism: a metric whose sign must be chosen after seeing the target performance is not a predictive transferability measure; it is a curve-fit to the benchmark. This is aggravated by the paper's own Friedman test: even in the in-domain dataset-transferability experiment (Table 2), the best average rank is not statistically distinguishable from NLEEP, LEEP, LogME, and PARC (CD = 2.792). So the remaining evidence for 'outperforms existing metrics' is a single non-significant best rank. The benchmark and the released code are valuable, which is why a full reject is not warranted, but the central claim must be revised and the sign of S_FU must be justified a priori or evaluated with held-out sign selection. This matches the reader's conditional verdict, so no adjustment is needed beyond the revisions already recommended.","tokens_in":20406,"tokens_out":9712,"duration_ms":93621,"concrete_test":"Recompute the Ours column of Table 3 using the original Eq. 5 normalization of S_FU in all cases, without applying Eq. 8 or any 1-S transformation. If the un-flipped Ours has a negative average weighted Kendall tau and does not rank first, the reported cross-domain superiority is a sign-selection artifact rather than a property of the proposed metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Ours outperforms existing metrics in cross-domain transfer (Abstract; Table 3) is not supported by a stable metric. In the cross-domain experiments, the authors observe that Ours, like PARC, NCTI, LEEP, and NLEEP, has predominantly negative weighted Kendall tau, and then apply Eq. 8, which reverses the S_FU normalization relative to Eq. 5. Since Kendall's tau is antisymmetric under score reversal, this is equivalent to selecting, after seeing the data, the orientation that yields positive correlation. The paper acknowledges that this 'contradicts intuition' and speculates about a gap in understanding, but this does not validate the proposed hypothesis that a larger conv2/conv1 gradient ratio indicates better adaptability. If the original sign had been kept, Ours would have a negative average tau across targets and would rank near the bottom of Table 3. The selective application of the 1-S transformation to some methods but not others (LogME, SFDA) makes the comparison non-principled. Thus the cross-domain half of the central claim is an artifact of orientation selection, and the remaining in-domain advantage is not statistically significant according to the paper's own Friedman test (p=0.002, CD=2.792, with NLEEP, LEEP, LogME, and PARC within the critical difference).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a transferability metric S = S_LP x S_FU for ranking pretrained models in medical image classification, where S_LP is an NCA-based label prediction probability and S_FU is the normalized ratio of conv2-to-conv1 gradient norms from a single triplet-loss backward pass. The authors introduce two evaluation scenarios: source-dataset transferability for 11 MedMNIST targets with 14 source datasets, and cross-domain model transferability for 9 ImageNet-pretrained architectures. They report that the proposed method achieves the best average weighted Kendall tau rank in both scenarios, and they release a large ground-truth benchmark from more than 20,000 fine-tuned models.","tokens_in":20666,"tokens_out":5502,"duration_ms":54763,"significance":"If the central claims were supported, the metric would have clear practical value: practitioners could select a source model with one forward and one backward pass instead of exhaustive fine-tuning. The paper also makes a useful empirical contribution by releasing ground-truth transfer performance tables for 15 source datasets, 9 architectures, and 11 medical targets, along with publicly available code. However, the cross-domain claim depends on a post hoc sign reversal of S_FU (Eq. 8 vs. Eq. 5), and the in-domain advantage is not statistically significant according to the paper's own Friedman test. The benchmarking data and the exploratory observation that cross-domain transfer from natural to medical images behaves differently remain valuable, but the stated conclusion of outperforming existing metrics in both settings is not supported as written.","major_comments":[{"comment":"The cross-domain experiment does not support the claim that the proposed metric outperforms existing methods. The authors first observe predominantly negative weighted Kendall tau for Ours (and for PARC, NCTI, LEEP, and NLEEP), and then apply Eq. (8), which reverses the S_FU normalization relative to Eq. (5). Because Kendall's tau is antisymmetric under score reversal, this is equivalent to selecting, after seeing the data, the orientation that yields positive correlation. With the original sign, Ours would have mostly negative tau values and would rank near the bottom of Table 3. The text acknowledges that the transformation 'contradicts intuition' but does not provide independent evidence for the hypothesized direction of S_FU. The Abstract's claim of outperformance in cross-domain transfer is therefore unsupported; the authors should either validate the sign on held-out targets before computing correlations or present the analysis as exploratory rather than as a metric comparison.","section":"Table 3 and Eq. (8)"},{"comment":"Even in the in-domain source-dataset scenario, the stated outperformance is not statistically supported. The Friedman test rejects the null hypothesis (p = 0.002), but with a critical difference of 2.792 and the best average rank 1.91, all methods with average rank up to 4.70 are within the critical difference. This includes NLEEP (3.45), LEEP (3.91), LogME (4.00), and PARC (4.00). The descriptive statement that Ours outperforms NLEEP on six targets is correct, but the general conclusion that the method 'outperforms existing transferability metrics' goes beyond what the statistical test supports. The sentence that additional datasets would make the difference significant is speculative and should not be used as evidence; either temper the claim or provide additional experiments.","section":"Table 2, Friedman test"},{"comment":"The gradient ratio S_FU is intended to make gradients comparable across source models, but in the cross-domain experiment the models differ not only in pretraining data but also in architecture: ResNet18, DenseNet121, EfficientNetV2-S, MobileNetV3-Small, GoogleNet, MnasNet-1.0, VGG11, ConvNeXt-Tiny, and ShuffleNetV2-0.5x have different first-layer kernel sizes, strides, channel widths, and depths. A raw ratio of conv2/conv1 gradient norms has no common scale across these architectures, and the paper does not justify why this ratio is comparable. This is a load-bearing issue for the cross-domain rankings, because the same metric is applied to models that are not directly comparable through a simple gradient-norm ratio.","section":"Section 3.2, Eq. (4), and Table 3"},{"comment":"The 1 - S transformation is applied selectively to PARC, NCTI, LEEP, NLEEP, and Ours, but not to LogME or SFDA. If the transformation is meant to correct for systematically negative correlations, it should be applied to all methods under a prespecified rule, or to none. Applying it to some methods but not others makes the comparison in Table 3 non-principled and can invert the relative ordering of methods. The authors should either justify a single decision rule for when the transformation is applied or compare all methods under the same convention.","section":"Section 5.2 and Table 3"}],"minor_comments":[{"comment":"There are several typographical errors, including 'Chaves el al.' (should be 'Chaves et al.') and 'transferability estation' (should be 'transferability estimation').","section":"Introduction and Related Work"},{"comment":"The paper uses two different min-max normalizations for S_FU with opposite ordering. Please define a single normalization convention and state explicitly how the cross-domain normalization in Eq. (8) is computed across models, since it is not the same as Eq. (5).","section":"Eqs. (5) and (8)"},{"comment":"SFDA has missing tau values on binary classification tasks, and the paper assigns the lowest rank for these missing values. This imputation should be justified, and a sensitivity analysis (e.g., excluding SFDA from the Friedman test) should be reported because it affects the average ranks.","section":"Table 2"},{"comment":"Please specify which source model and which target subset are shown in the t-SNE projection. The caption currently only says 'binary Pneumonia classification,' so the reader cannot determine how representative the figure is.","section":"Figure 3"},{"comment":"For targets where the proposed method's predictions are negatively correlated, the ground-truth AUC differences between the best and worst source models are very small (e.g., 0.004 for Blood and 0.007 for Path). The paper should discuss whether the correlation values are practically meaningful in such compressed ranges, because small ranking differences may not correspond to clinically relevant performance differences.","section":"Section 5.2, Table 3"},{"comment":"The final score is a product of two normalized terms, but the choice of product over sum is not justified. Since S_LP values can be very small, the product may be dominated by one term; a sensitivity analysis using an additive combination or different normalization would be informative.","section":"Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The authors are transparent about the Friedman non-significance and about the 'contradicts intuition' sign reversal, but these admissions are not reflected in the Abstract and Conclusions. The cross-domain orientation selection is a serious issue that may be fixable if the authors either validate the sign on held-out targets or remove the cross-domain outperformance claim and reframe the contribution as a benchmark and an exploratory analysis. Given the value of the released ground-truth data, I see a path to revision rather than a categorical rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for the benchmark and the NCA idea, but the cross-domain half of the central claim does not survive contact with its own Table 3. The in-domain result is genuinely interesting though not statistically decisive.\n\nWhat's new: a transferability score combining NCA-based feature suitability with a conv2/conv1 gradient ratio from a single backward pass, evaluated in two scenarios that medical imaging hasn't systematically benchmarked. The ground-truth tables (15 sources, 9 architectures, 11 MedMNIST targets, 20k+ trained models) are a real resource, and the paper is honest about hyperparameter search and early stopping. The NCA motivation against regularized FDA is well argued, and the ablation shows the gradient term helps on several targets.\n\nSoft spots: First, in Table 2, the Friedman test rejects the null but the critical difference is 2.792, and NLEEP, LEEP, LogME, and PARC fall within it. So \"outperforms existing metrics\" is not supported on the in-domain scenario; \"best average rank\" is. Second, the cross-domain scenario is the bigger problem. The paper applies a 1-S transformation to PARC, NCTI, LEEP, and NLEEP, and an equivalent sign flip for its own S_FU via Eq. 8. Since Kendall's tau is antisymmetric under score reversal, this is selecting the orientation after seeing the ground-truth correlations. Without the flip, Ours has predominantly negative tau and ranks near the bottom. The paper notes this \"contradicts intuition\" and invokes a gap in understanding, but that doesn't make the reported rankings a valid test of the proposed hypothesis. The selective transformation (not applied to LogME and SFDA) makes the comparison non-principled. The Friedman test also shows no significant difference in this scenario, yet the abstract claims outperformance \"in both settings.\"\n\nWhat holds up: the benchmark data, the NCA idea, and the empirical observation that medical sources can beat ImageNet on some targets. Those are worth publishing. What needs fixing: either pre-register the sign of S_FU, or present the cross-domain experiment as exploratory and drop the superiority claim. The authors should also add repeated runs with error bars and clarify the SFDA rank handling.\n\nI'd send this to peer review, because the resource and the in-domain idea deserve referee time, but the revision needs to be substantial. It's a solid conditional accept for a workshop-type venue, or major revision for a journal.","headline":"A valuable benchmark and a promising NCA-based metric, but the cross-domain superiority claim is an artifact of post hoc sign reversal; the in-domain advantage is not statistically significant.","tokens_in":21252,"tokens_out":2225,"would_cite":true,"duration_ms":21595,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a transferability score combining NCA label-prediction confidence with the ratio of conv2 to conv1 gradient magnitudes, and shows it ranks medical image sources better than existing feature-only metrics.","keywords":["transfer learning","medical image classification","transferability estimation","dataset similarity","MedMNIST","neighborhood component analysis","gradient-based ranking","model selection"],"falsifier":"Run the score on a held-out medical target outside MedMNIST and compare the rank correlation of the stated $S_{FU}$ with its sign-flipped version; if the sign-flipped version systematically ranks better, the fixed-direction premise is false.","tokens_in":20167,"feed_emoji":"🩺","tokens_out":9880,"duration_ms":85470,"temperature":0.7,"pith_summary":"This paper tries to establish that choosing a pretrained source model for a medical image classification task should depend not only on how well the source's features already separate the target classes, but also on how easily those features can adapt to the target's subtle, texture-driven patterns. It proposes a transferability score $S = S_{LP} \\times S_{FU}$ that multiplies a label-prediction probability computed after a Neighborhood Component Analysis projection of the target features (feature suitability) by the ratio of gradient norms in the second and first convolutional layers from a single triplet-loss backward pass (feature adaptability). Across 11 MedMNIST targets with 14 source datasets, the score ranks sources in closer agreement with true fine-tuned AUC than LogME, SFDA, PARC, NCTI, LEEP, and NLEEP, reaching the best average rank in both the source-dataset and cross-domain model-selection scenarios. The paper also reports that source dataset size alone does not predict transfer performance, that source diversity matters more than similarity, and that medical-specific sources beat ImageNet on 4 of 11 targets. A practitioner who trusts the result can skip exhaustive fine-tuning and select a source with one forward pass and one backward pass.","feed_headline":"Gradient-aware score beats feature-only metrics for medical transfer","feed_subtitle":"A fast forward and backward pass can pick the best pretrained source, replacing exhaustive fine-tuning.","key_machinery":"The central object is the composite transferability score $S = S_{LP} \\times S_{FU}$. $S_{LP}$ uses Neighborhood Component Analysis on the penultimate-layer features of the source model to approximate fine-tuning dynamics without inverting a covariance matrix, producing a label-prediction probability via a 5-NN classifier. $S_{FU}$ is the ratio of the $\\ell^2$ norms of gradients with respect to the second and first convolutional layers, computed from a single backward pass of a triplet margin loss; the first layer acts as a normalizer because it is expected to change little, while the second indicates how much the source would adapt. The product is meant to reward sources whose features are both already separable and able to update toward new local texture patterns, and the final ranking is evaluated against true fine-tuned AUC using weighted Kendall's $\\tau_w$.","core_discovery":"The central claim is that transferability for medical image classification decomposes into two measurable parts: suitability and adaptability. Suitability is captured by $S_{LP}$, the summed probability that a 5-nearest-neighbor classifier assigns the target labels after the source model's penultimate features are projected by Neighborhood Component Analysis, which mimics the class-separating effect of fine-tuning without matrix inversion. Adaptability is captured by $S_{FU}$, the ratio $\\|\\nabla_{\\theta_{conv2}}L\\|_2 / \\|\\nabla_{\\theta_{conv1}}L\\|_2$ obtained from one backward pass of a triplet margin loss through the source model; the paper argues a larger ratio means the source can more readily learn new local patterns. The product of these normalized terms, $S = S_{LP} \\times S_{FU}$, is claimed to rank source datasets better than feature-only transferability metrics, and to avoid their self-source bias whereby a model pretrained on the target itself looks like the best source. The paper further claims the same score, with the gradient term's sign reversed, ranks ImageNet-pretrained architectures for cross-domain transfer to medical targets, and interprets the sign inversion as evidence that cross-domain transfer dynamics differ fundamentally from in-domain transfer.","pith_inferences":["The paper leaves open whether the sign of the gradient term is a stable property or an artifact of the setting; a natural extension is a domain-aware score that learns the sign of $S_{FU}$ from a small validation set rather than fixing it.","The released ground-truth benchmark (15 sources, 9 architectures, 11 targets) could be reused to train a lightweight rank learner that treats $S_{LP}$ and $S_{FU}$ as features, which the paper does not attempt.","The NCA approximation of fine-tuning may carry over to other low-data domains where feature dimension exceeds sample size, but the paper only demonstrates it on MedMNIST.","The negative correlation between feature suitability and transfer in cross-domain settings suggests suitability metrics may be rewarding shortcut alignment rather than true adaptability; a direct test would compare $S_{LP}$ with shortcut measures on the fine-tuned models."],"forward_implications":["Source model selection for a medical target can be reduced to one forward pass and one backward pass, avoiding grid-searched fine-tuning of every candidate.","Public medical datasets, alone or combined as a leave-target-out set, can outperform ImageNet as transfer sources on 4 of 11 targets, so ImageNet should not be the default.","Dataset size is not a reliable guide: a small source like Breast can beat a much larger source like OrganS on most targets, so transferable sources must be ranked by content rather than scale.","Existing feature-quality transferability metrics suffer from self-source bias and weak or negative correlations on medical targets, while the proposed combined score maintains positive correlation across all in-domain targets.","Cross-domain transfer from natural to medical images appears to invert the adaptability signal: better transfer is associated with smaller early-layer updates and larger prediction deviations, pointing to a different mechanism than in-domain transfer."],"supporting_citations":[{"why":"Shows existing transferability metrics fail to generalize to medical tasks, motivating a medical-specific metric.","marker":"[10]"},{"why":"Supplies the MedMNIST target datasets and the pretrained ResNet18 source models used in both benchmarking scenarios.","marker":"[11, 12]"},{"why":"LEEP is a feature-suitability baseline whose self-source bias the paper aims to fix.","marker":"[17]"},{"why":"NLEEP is the strongest feature-based baseline and the main runner-up in the dataset-transfer experiment.","marker":"[18]"},{"why":"LogME is a baseline that adds a Bayesian linear model to target features.","marker":"[21]"},{"why":"SFDA is the Fisher-discriminant baseline whose low-data failure motivates the NCA component.","marker":"[22]"},{"why":"NCTI is a neural-collapse-based baseline compared in both transferability scenarios.","marker":"[23]"},{"why":"Defines weighted Kendall's tau, the ranking-correlation measure used to evaluate all transferability scores.","marker":"[29]"},{"why":"Neighborhood Component Analysis is the mechanism behind the label-prediction suitability term.","marker":"[31]"},{"why":"The triplet margin loss is the loss whose backward pass produces the conv2/conv1 gradient ratio.","marker":"[33]"}],"fun_headline_variants":["Gradient-aware transfer score beats feature-only for medical","Medical transfer: gradients + features improve source ranking","Avoid self-source bias with dual-part transferability metric","Cross-domain medical transfer improved by gradient metric"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking depends on the conv2/conv1 gradient ratio having a fixed sign, with larger early-layer updates meaning better transfer, yet the cross-domain experiment requires the opposite sign to achieve positive correlations.","fun_headline_variants_meta":{"raw":{"variants":["Gradient-aware transfer score beats feature-only for medical","Medical transfer: gradients + features improve source ranking","Avoid self-source bias with dual-part transferability metric","Cross-domain medical transfer improved by gradient metric"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1327,"prompt_tokens":966,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":582,"tokens_out":361,"duration_ms":4328,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:29:14.332062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the score on a held-out medical target outside MedMNIST and compare the rank correlation of the stated $S_{FU}$ with its sign-flipped version; if the sign-flipped version systematically ranks better, the fixed-direction premise is false.","supporting_citations":[{"cited_title":"Chaves, A","cited_arxiv_id":null,"evidence_quote":"Shows existing transferability metrics fail to generalize to medical tasks, motivating a medical-specific metric."},{"cited_title":"Nguyen, T","cited_arxiv_id":null,"evidence_quote":"LEEP is a feature-suitability baseline whose self-source bias the paper aims to fix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NLEEP is the strongest feature-based baseline and the main runner-up in the dataset-transfer experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LogME is a baseline that adds a Bayesian linear model to target features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SFDA is the Fisher-discriminant baseline whose low-data failure motivates the NCA component."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NCTI is a neural-collapse-based baseline compared in both transferability scenarios."},{"cited_title":"Vigna, A weighted correlation index for rankings with ties, in: Pro- ceedings of the 24th international conference on World Wide Web, 2015, pp","cited_arxiv_id":null,"evidence_quote":"Defines weighted Kendall's tau, the ranking-correlation measure used to evaluate all transferability scores."},{"cited_title":"Goldberger, G","cited_arxiv_id":null,"evidence_quote":"Neighborhood Component Analysis is the mechanism behind the label-prediction suitability term."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The triplet margin loss is the loss whose backward pass produces the conv2/conv1 gradient ratio."}],"review_version":1}