{"id":"aafa2c0c-1d1c-4f3c-9cad-1fad88ddf425","arxiv_id":"1908.07630","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A single weighted combination of log dataset size and feature-space distance, tuned on nine tasks, selects the best transfer source model in 13 of 21 held-out image and semantic relation tasks.","lead":"The paper introduces P2L, a method that predicts which pretrained model will transfer best to a new learning task by combining the source dataset's size with the distance between source and target datasets in a reference network's feature space. It reports that P2L picks the best source model in 13 of 21 image and semantic relation tasks, beating the common heuristic of just picking the largest dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The distance proxy D may be biased by the reference model VGG16 being pretrained on ImageNet1K, which is both a candidate source and semantically overlapping with the target datasets; without a disjoint reference-model control, the reported ranking success may reflect this overlap rather than…","rationale":"The reader's weakest assumption correctly identifies the distance proxy D as the load-bearing element of the central claim. My stress-test sharpens this by pointing to a concrete confound: the specific reference model used for images, VGG16 pretrained on ImageNet1K, is not an arbitrary feature extractor. It is one of the candidate source models and its pretraining classes overlap with several target datasets. This creates a favorable bias in D that the paper does not control for. The empirical Spearman correlations and 13/21 success count are therefore not sufficient to establish that D reflects transferability in general. A single control experiment with a disjoint reference model would settle whether the method's success is an artifact of the ImageNet-pretrained space. Because this is a missing control rather than an observed failure, the appropriate outcome is conditional acceptance: the claim is plausible but requires the additional validation. This is consistent with the reader's CONDITIONAL verdict, though the specific emphasis differs, hence partial agreement.","tokens_in":11290,"tokens_out":20888,"duration_ms":225563,"concrete_test":"Re-run the 14 image experiments with D(t_i,s_j) computed using a reference model pretrained on a non-overlapping domain (e.g., VGG16 pretrained on Places365, or a ResNet-50 pretrained on a non-image dataset), while keeping the source/target fine-tuning and evaluation identical. Compare the number of tasks where P2L selects the true best source and the Spearman correlations against the VGG16-ImageNet1K results. If the best-source count drops materially (e.g., from 9/14 toward the size-only baseline) or the correlations fall toward 0, the reported performance is confounded by the reference model's pretraining overlap; if the ranking quality is preserved, the concern is answered.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that E(t_i,s_j) = z(log(|s_j|)) + k·z(D(t_i,s_j)) accurately ranks transfer sources depends on D being a reliable proxy for transfer improvement. In Section 3.2, D is computed in the feature space of VGG16 pretrained on ImageNet1K. ImageNet1K is itself one of the 17 candidate sources (Section 4.1), and several target datasets (CUBS, Cars, Oxford, Wikiart) contain object categories that overlap ImageNet1K's class distribution. The penultimate-layer features of VGG16 are therefore already optimized to separate exactly the categories that distinguish these sources and targets, which can make D(t_i, ImageNet1K) artificially small and inflate P2L's ranking success without measuring true transferability of the source weights. The paper reports only correlations on tasks where this confound is present and provides no control experiment with a reference model trained on a domain disjoint from the candidate sources and targets. If the success is an artifact of the ImageNet-pretrained reference space, the method will not generalize to new domains where such a matched reference model is unavailable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes P2L (Predict To Learn), an estimator E(t_i, s_j) = z(log(|s_j|)) + k·z(D(t_i, s_j)) that ranks candidate source models for transfer learning to a target task, using only the source dataset size and a feature-space distance between target and source datasets. The distance is computed by passing each dataset through a fixed reference model (VGG16 pretrained on ImageNet1K for images, a PCNN for semantic relations) and comparing the mean penultimate-layer activations. The parameter k is tuned on 9 ImageNet22K-derived tasks and then evaluated on 14 image tasks and 7 semantic-relation tasks, reporting Spearman correlations of 0.703 and 0.763 and an average accuracy of 67.22% versus 64.47% for the largest-dataset baseline and 64.86% for the least-divergent baseline (Table 1). The paper also presents an experiment on merged source datasets showing that a larger merged dataset does not always transfer better than ImageNet1K.","tokens_in":11562,"tokens_out":8807,"duration_ms":264100,"significance":"If the proposed estimator is a reliable predictor of transferability, it offers a practical and inexpensive way to select among pre-trained source models, which is of direct use for cloud-based training services. The held-out evaluation design—tuning k on one set of tasks and evaluating on disjoint image and relation tasks—is a genuine strength, as is the cross-domain transfer of k from images to relations. The merged-dataset experiment provides a useful counterexample to the assumption that larger sources are always better. However, the evidence is weakened by the possible confound in the distance measure, the lack of statistical significance reporting, and some presentation issues that affect the comparability of the headline numbers.","major_comments":[{"comment":"The distance D(t_i, s_j) is computed in the feature space of VGG16 pretrained on ImageNet1K, which is itself one of the 17 candidate source datasets listed in Section 4.1. Several target datasets (CUBS, Cars, Oxford, Wikiart) contain object categories that overlap ImageNet1K's class distribution. The penultimate-layer features of VGG16 are therefore optimized to separate exactly the categories that distinguish these sources and targets, which can make D(t, ImageNet1K) artificially small and inflate P2L's ranking accuracy without measuring true transferability. The paper provides no control experiment with a reference model trained on a domain disjoint from the candidate sources and targets. Without such a control, the reported Spearman correlations (Table 2 and Figure 6A) may reflect this overlap rather than a general property of the feature space. Please add an experiment using a reference model trained on a disjoint dataset (or a randomly initialized network) to demonstrate that the method is not reliant on the source pool being matched to the reference model's training data.","section":"Section 3.2 and Section 4.1"},{"comment":"The comparisons in Table 1 rely on point estimates with no confidence intervals or significance tests. Per-target Spearman rho values in Table 2 range from 0.52 to 0.95, and with only 17 sources per target, the uncertainty in each correlation is substantial. For semantic relations, the reported gain of 71.79 versus 70.6 for the largest-dataset baseline is small and may be within noise. To support the claim that P2L outperforms the baselines, please report task-level error bars or confidence intervals, and where meaningful, paired statistical tests across the 21 tasks.","section":"Section 4.3, Tables 1-4"},{"comment":"The abstract states that P2L was validated 'across 95 tasks' and that the largest-dataset baseline selected the best model in only 55 cases, whereas the body evaluates 21 tasks and reports that P2L picked the best model in 13 cases; this discrepancy must be resolved. Additionally, Table 1 labels all values as 'Mean Top-1 Accuracy' even though the semantic-relations results are area under the PR curve, and the 'Average over 21 tasks' row averages raw values from two different metrics, which is not meaningful. Please report the two domains separately with their appropriate metrics and either remove the combined average or justify it explicitly.","section":"Abstract and Table 1"}],"minor_comments":[{"comment":"The paper says the estimator form was chosen 'based on extensive experimentation' but does not report the tuned value of k or the range over which it was searched; reporting these would improve reproducibility.","section":"Section 3.1 and Section 4.3.1"},{"comment":"The description of how target datasets are created from ImageNet22K partitions is ambiguous: 'One-tenth of the third partition was used to create a transfer learning target' does not specify the resulting number of training images per target task. Please clarify the exact construction procedure.","section":"Section 4.1"},{"comment":"There are typos in the Future Work section, including 'oc more enriched relationships' and 'may be provide'; the manuscript would benefit from a careful proofreading pass.","section":"Section 5"},{"comment":"The merged-dataset experiment with LC and ImageNet1K is interesting but is not explicitly connected to the P2L evaluation; state how this observation supports the main claim about the importance of both size and divergence.","section":"Section 4.3.3"},{"comment":"The caption for Figure 5 is incomplete ('Log10(Size) of sourcesTv'); please provide a full caption describing the contents.","section":"Figure 5"},{"comment":"The column header 'Least Divergent' refers to baseline B5, which uses the distance D; clarify this definition in the table caption or in the text.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The reference-model circularity is the main technical risk: if the distance measure's success depends on the reference model being pretrained on one of the candidate sources, the method's generality is substantially weaker than claimed. If the authors can provide a control with a disjoint reference model and improve the statistical reporting, the paper could be acceptable. The discrepancy between the abstract's '95 tasks' and the body's '21 tasks' should be fixed before any final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper is a modest but credible empirical study of a simple transfer-source-selection estimator. The estimator is z(log source size) + k·z(feature-space distance), with k tuned once on a separate set of 9 ImageNet-derived tasks. That held-out design is a real step up from the usual fitting-to-the-evaluation practice in this area. The evaluation spans 14 image tasks and 7 semantic-relation tasks, and the results are reported as Spearman correlations (0.70 and 0.76) and a 62% hit rate for picking the best source across 21 tasks. Those numbers are plausible, and the two-domain validation is genuinely useful.\n\nWhat is new here is not the components—size and similarity have both been used—but the combination in a simple linear predictor with one learned weight, and the demonstration that it works across images and relations. The paper is also honest that it is an engineering heuristic, not a theory.\n\nNow the soft spots. The stress-test concern is legitimate: the image reference model is VGG16 trained on ImageNet1K, which is itself a candidate source and is semantically close to several of the target datasets (CUBS, Cars, Oxford, Wikiart). So the distance to ImageNet1K may be artificially small, and the ranking success may partly reflect that the reference model is already matched to the source set. There is no control experiment with a reference model whose training domain is disjoint from the candidate sources. I would not call this fatal, because for many deployed applications you do have a reference model trained on common data, but it does limit the generality claim.\n\nTwo other issues: first, the abstract reports 95 tasks while the body says 21; the 71 cloud-API tasks are a separate, weaker analysis and should not be conflated. Second, there are no error bars or significance tests, and the gains over the largest-dataset baseline are small—about 2.7 points on average, and only 1.2 points for relations. Without uncertainty estimates, it's hard to tell if the distance term adds real signal beyond size alone. No code or data are released either.\n\nOverall, the central argument holds up in the narrow sense: the estimator does rank sources better than the two simple baselines on these tasks. The magnitude of the improvement is uncertain, and the reference-model confound deserves attention. This is a paper for transfer-learning practitioners who want a cheap source-selection method and for researchers working on task similarity. I would send it to a serious referee, with a request for significance tests and a reference-model control. I'd probably cite it if I worked in this niche, but with a caveat about the confound.","headline":"A held-out-tested estimator for transfer source selection, with a reference-model confound that limits but does not destroy the claim.","tokens_in":12089,"tokens_out":4776,"would_cite":true,"duration_ms":488150,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cheap score picks best transfer source in 13 of 21 tasks","keywords":["transfer learning","source model selection","fine-tuning","embedding divergence","dataset size","feature space distance","semantic relation prediction","image classification"],"falsifier":"Run P2L on a set of source tasks that are entirely absent from the reference model's training data, exhaustively fine-tune all sources on the targets, and compare the predicted and true rankings; if the Spearman correlation drops to near zero, the distance signal is an artifact of the reference model rather than a general transfer property.","tokens_in":11109,"feed_emoji":"🎯","tokens_out":9146,"duration_ms":78093,"temperature":0.7,"pith_summary":"The paper asks a practical question: when several pre-trained source models are available, which one should be fine-tuned for a new target task? It proposes an estimator, called P2L (Predict To Learn), that ranks sources using only the log-size of the source dataset plus a feature-space distance between source and target, computed with a single forward pass through one reference network. On the paper's 21-task validation, covering image classification and semantic-relation prediction, P2L selects the best source in 13 tasks and reaches 67.22% mean accuracy, versus 64.47% for the largest-dataset baseline and 64.86% for the least-divergent baseline. If these results hold, transfer-learning source selection no longer requires exhaustively fine-tuning every candidate.","feed_headline":"Cheap score picks best transfer source in 13 of 21 tasks","feed_subtitle":"P2L uses log dataset size plus feature distance, beating largest-dataset and closest-dataset baselines.","key_machinery":"The load-bearing object is the estimator $E(t_i,s_j)$ itself. Each dataset is reduced to a single summary vector: the mean of penultimate-layer feature activations from a reference model (VGG16 for images, PCNN for relations), L1-normalized. $D(t_i,s_j)$ is then a divergence between those summaries, with KL and cityblock working well; $\\log(|s_j|)$ encodes the empirically saturating benefit of larger sources; the $z$-scoring makes the two terms comparable; and $k$, tuned once on a small collection of tasks, balances them. Because the target needs only one forward pass through the reference network, the whole ranking costs seconds to minutes rather than a training run per candidate.","core_discovery":"The central claim is that transfer improvement $I(t_i,s_j)=P(M(t_i,s_j))-P(M(t_i,\\phi))$ is ranked accurately by the embedding-divergence estimator $E(t_i,s_j)=z(\\log(|s_j|))+k\\,z(D(t_i,s_j))$, where $|s_j|$ is source dataset size, $D$ is a distance between dataset summaries in the penultimate layer of a fixed reference network, and $k$ is a single tuned scalar. The paper validates this by fine-tuning every source-target pair for 14 image tasks and 7 semantic-relation tasks, and reports Spearman correlations of 0.707 and 0.763 between predicted and true rankings in the two domains. It also reports that merging source datasets (ImageNet22K plus Places2) does not reliably beat the better individual source, which supports the view that transfer depends on compositional match, not just total data.","pith_inferences":["A natural extension is to pre-compute summary vectors for every model in a model zoo, turning source selection into a nearest-neighbor lookup in feature-vector space; the paper's setup already supplies the once-per-source computation.","Because the distance term is measured through one reference network, the method's reliability may depend on that reference; a testable variant would average the ranking over several reference networks trained on different data before trusting the recommendation.","The merged-dataset result suggests a counterintuitive practical rule: when expanding training data, adding examples from unrelated categories can reduce transfer value even as dataset size grows; this is an inference beyond the paper's experiments, but consistent with its size-versus-divergence trade-off."],"forward_implications":["A cloud service can recommend a fine-tuning source for a new customer dataset in about a minute per thousand images rather than training every candidate model.","The largest-dataset heuristic is not safe: across 21 tasks P2L outperforms it in mean accuracy and in the number of times the best source is selected.","The same estimator and the same tuned $k$ transfer across domains, working for both image classification and semantic-relation prediction.","The best source is usually near the top of P2L's ranking: the paper reports that finding it required at most three P2L picks for all 21 tasks, versus six for the largest-dataset method and five for the least-divergent method."],"supporting_citations":[{"why":"Supplies the VGG16 ImageNet reference model and the common assumption that the model trained on the largest dataset is the best transfer source.","marker":"[28]"},{"why":"Gives the empirical log-scaling of accuracy with training-set size that motivates the log-size term in the estimator.","marker":"[15]"},{"why":"Documents negative transfer from ill-chosen sources, the failure mode that a source-prediction method must avoid.","marker":"[29]"},{"why":"Contributes nine of the visual target tasks used in the main validation.","marker":"[1]"},{"why":"Provides the CC-DBP knowledge-base corpus from which the semantic-relation sources and targets are drawn.","marker":"[14]"},{"why":"Supplies the PCNN architecture used as the reference model in the relation-prediction experiments.","marker":"[40]"},{"why":"Defines a prior learning-to-transfer approach that requires more curated experience and is the main contrast for P2L's single-reference estimate.","marker":"[37]"},{"why":"Offers task2vec, a low-cost task-embedding method that P2L positions itself against on compute-versus-performance trade-offs.","marker":"[2]"},{"why":"Provides the ImageNet22K hierarchy from which 15 image source and target datasets are carved.","marker":"[16]"}],"fun_headline_variants":["P2L predicts best transfer model via embedding distance","Cheap estimator ranks transfer sources better than data size","P2L chooses transfer source by feature overlap, not volume","Predict transfer success: P2L beats largest-dataset heuristic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking collapses if the distance between source and target summaries in one fixed reference model's feature space is not a reliable proxy for fine-tuning improvement, especially since that reference model is trained on one of the candidate sources (ImageNet1K for images).","fun_headline_variants_meta":{"raw":{"variants":["P2L predicts best transfer model via embedding distance","Cheap estimator ranks transfer sources better than data size","P2L chooses transfer source by feature overlap, not volume","Predict transfer success: P2L beats largest-dataset heuristic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2074,"prompt_tokens":940,"completion_tokens":1134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":1067}},"tokens_in":556,"tokens_out":1134,"duration_ms":9593,"temperature":1.0,"reasoning_tokens":1067,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:01:18.649136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run P2L on a set of source tasks that are entirely absent from the reference model's training data, exhaustively fine-tune all sources on the targets, and compare the predicted and true rankings; if the Spearman correlation drops to near zero, the distance signal is an artifact of the reference model rather than a general transfer property.","supporting_citations":[{"cited_title":"CNN Features off-the-shelf: an Astounding Baseline for Recognition","cited_arxiv_id":"1403.6382","evidence_quote":"Supplies the VGG16 ImageNet reference model and the common assumption that the model trained on the largest dataset is the best transfer source."},{"cited_title":"To transfer or not to transfer","cited_arxiv_id":null,"evidence_quote":"Documents negative transfer from ill-chosen sources, the failure mode that a source-prediction method must avoid."},{"cited_title":"Zitnick K","cited_arxiv_id":null,"evidence_quote":"Contributes nine of the visual target tasks used in the main validation."},{"cited_title":"A Dataset for Web- scale Knowledge Base Population","cited_arxiv_id":null,"evidence_quote":"Provides the CC-DBP knowledge-base corpus from which the semantic-relation sources and targets are drawn."},{"cited_title":"Distant supervision for relation extraction via piece- wise convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the PCNN architecture used as the reference model in the relation-prediction experiments."},{"cited_title":"Transfer learning via learning to transfer","cited_arxiv_id":null,"evidence_quote":"Defines a prior learning-to-transfer approach that requires more curated experience and is the main contrast for P2L's single-reference estimate."},{"cited_title":"Socher L.-J","cited_arxiv_id":null,"evidence_quote":"Provides the ImageNet22K hierarchy from which 15 image source and target datasets are carved."}],"review_version":1}