{"id":"b5477182-1e44-40f2-bb06-7f8e2184dc7f","arxiv_id":"2608.09091","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A method and two metrics aim to predict which pretraining dataset transfers best to a target image classification task, validated only on plant leaf classification.","lead":"TLDChoiceNet is a neural network that predicts how well a fine-tuned image classifier will perform depending on which dataset was used for pretraining. The paper also introduces two statistical metrics that correlate with transfer learning success, but tests everything on a single fine-tuning task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is empirically ungrounded: all headline results (MSE, DD, ACC) come from a single fine-tuning dataset, and the paper states that the method degrades sharply on unseen fine-tuning datasets, so the key condition for the claim is not met.","rationale":"The reader's verdict and weakest assumption pin the paper's central failure on single-fine-tuning-dataset evaluation, and the manuscript text confirms this in Section 4.1 and Section 5.3. The paper's own honesty about the limitation is creditable, but it does not repair the gap: the headline MSE and R2 numbers are computed only on a test split from the same fine-tuning dataset as training, so they do not support the advertised 'given a fine-tuning dataset' generalization. The DD and ACC regressions are likewise computed on the same 9 TLDS points (3 similar, 6 dissimilar) and are not tested on a new target task. Because the central claim requires the method to work across fine-tuning datasets, and the only available evidence on that point shows failure (v1 MSE 0.46, 80% errors), the rejection is justified. The concrete test I propose is a multi-task holdout evaluation; if the method were shown to retain low MSE and high R2 on unseen fine-tuning datasets, the paper would merit a conditional acceptance, but as it stands, the load-bearing assumption is unverified and contradicted by the one available datapoint.","tokens_in":8114,"tokens_out":1462,"duration_ms":12915,"concrete_test":"Hold out one or more complete unseen fine-tuning datasets (e.g., a second plant dataset, a different image domain such as birds or cars) and recompute the three headline numbers: TLDChoiceNet v2 MSE, DD R2, and ACC R2 on those unseen tasks. If the MSE rises to the ~0.46 regime or the R2 values drop below, say, 0.5, the central claim fails as stated. A minimum acceptable check would be to report the full TLDS construction with more than one fine-tuning dataset and show that the DD/ACC regressions hold across at least two or three target datasets.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that TLDChoiceNet can predict post-fine-tuning test accuracy for a given fine-tuning dataset and candidate transfer dataset, and that DD/ACC can choose the optimal transfer dataset. For this to hold, the method must be able to generalize to unseen fine-tuning datasets. The paper's own stated experiments directly contradict this: Section 4.1 says that when testing with an unseen fine-tuning dataset, TLDChoiceNet v1's MSE rises to 0.46 and predictions are off by up to 80%. The reported headline MSE of 0.031 (v2) and R2 values of 0.894/0.974 are all computed on a test split that shares the same fine-tuning dataset as training, as the paper notes: 'the test data shared the fine-tuning dataset with the training data.' There is no evaluation of DD or ACC on an unseen fine-tuning dataset, and Section 5.3 explicitly lists testing with multiple fine-tuning datasets as future work. The strongest performance numbers are therefore in-sample fits on nine TLDS points, which is insufficient evidence for a 'given a fine-tuning dataset' method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TLDChoiceNet, a network that takes a fine-tuning dataset and a candidate transfer-learning dataset (or its transfer-learned model) and predicts the post-fine-tuning test-set accuracy. Version 1 uses a PCA-based embedding and reports 0.154 MSE; version 2 uses per-class ImageNet-pretrained ResNet50 v2 activations for the fine-tuning dataset and per-class TransferModel activations for the transfer dataset, reporting 0.031 MSE. The paper also proposes two unsupervised metrics, distribution distance (DD) and average class correlation (ACC), and reports linear-regression R^2 values of 0.894 and 0.974 against fine-tune accuracy. All experiments use one fine-tuning dataset (22-class plant leaves), and the paper explicitly states that the reported test split shares the fine-tuning dataset with training. When tested on an unseen fine-tuning dataset, TLDChoiceNet v1 degrades to MSE 0.46 with predictions off by up to 80%. The authors list testing with multiple fine-tuning datasets as future work.","tokens_in":8381,"tokens_out":6840,"duration_ms":71761,"significance":"If the central claim held, the method would give practitioners a way to select a pretraining dataset without fine-tuning every candidate, and the DD/ACC metrics would provide a cheap unsupervised proxy. The paper has useful ingredients: a clearly specified algorithm, a created dataset of transfer-learning tuples, and a public code link. However, the current empirical support is far too narrow for the stated claim. The headline numbers come from a single target task, the DD/ACC regressions are in-sample fits on nine points, and the one out-of-domain fine-tuning result reported is a large degradation. The paper is honest about these limitations, but the limitations directly undercut the abstract's promise of a general method.","major_comments":[{"comment":"The central claim that TLDChoiceNet chooses the best transfer learning dataset 'given a fine tuning dataset' is not supported because all reported TLDChoiceNet results use the same 22-class plant-leaves fine-tuning dataset. The only result with an unseen fine-tuning dataset, reported in Section 4.1, is v1's MSE of 0.46 with predictions off by up to 80%; no analogous result is given for v2, DD, or ACC. Section 5.3 explicitly lists testing with multiple fine-tuning datasets as future work, so the current evidence does not establish the claimed generality.","section":"Section 4.1 and Section 5.3"},{"comment":"The DD and ACC R^2 values of 0.894 and 0.974 are computed from linear regressions fit to the same nine TLDS points that are used to report R^2. These are in-sample goodness-of-fit values, not predictive validations. With nine points split into a similar cluster and a dissimilar cluster, a two-group linear fit can produce a high R^2 without demonstrating that the metric discriminates among datasets within either group. The paper needs held-out validation or cross-validation, and ideally multiple target tasks, before claiming that DD or ACC can 'choose' a transfer dataset.","section":"Section 4.2, Tables 2 and 3"},{"comment":"The DD and ACC regressions ignore most of the TLDS. Tables 2 and 3 list only similar (three rows) and dissimilar (six rows) transfer-learning datasets, whereas Section 3.2 states that the TLDS also includes 40 random subsets from CIFAR-100 and ImageNet and a no-transfer condition. The random subsets are exactly the kind of candidates a general dataset-choice method must handle, so the reported correlations do not cover the claimed scope of the method. The paper should either include these cases in the regressions or justify their exclusion.","section":"Section 3.2 and Section 4.2"},{"comment":"The paper does not compare DD or ACC against simple baselines, such as predicting accuracy from the transfer dataset's source category, dataset size, or the no-transfer accuracy. Given that the reported tables contain only two broad categories (plant diseases and bird species), a trivial rule that assigns a high accuracy to similar datasets and a low accuracy to dissimilar datasets could reproduce much of the observed separation. A quantitative comparison to such baselines is necessary to support the claim that low-level statistics and class correlations are the explanatory factors.","section":"Section 4.2 and Tables 2/3"}],"minor_comments":[{"comment":"The text states that the average ACC for dissimilar TL datasets is 0.276, but the six values in Table 3 average to approximately 0.271; the discrepancy should be corrected or the averaging procedure clarified.","section":"Section 4.2"},{"comment":"References [9] and [10] are duplicate entries for the same Krizhevsky technical report; one of them should be removed or differentiated.","section":"References"},{"comment":"The paper reports training on 94 datapoints and shows test predictions in Figure 5, but it never states the number of test datapoints or the total TLDS size, which makes the MSE values difficult to interpret statistically. The test set size should be reported in the text.","section":"Section 4.1 and Figure 5"},{"comment":"Section 3.2 says the similar and dissimilar TL datasets each have three randomly sampled subsets, but Tables 2 and 3 list three similar rows and six dissimilar rows; this inconsistency should be resolved.","section":"Section 3.2"},{"comment":"The phrase 'MSE loss' is redundant; 'MSE' is sufficient.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent about its limitations, which is commendable, but those limitations directly undercut the abstract's claims. The single fine-tuning dataset, the in-sample DD/ACC regressions, and the reported v1 failure on an unseen fine-tuning dataset are load-bearing issues. I recommend inviting a major revision that adds multi-dataset evaluation and proper predictive validation of DD/ACC; if the authors cannot supply such experiments, rejection should be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a genuinely useful idea: predict which transfer learning dataset will give the best fine-tuned accuracy without fine-tuning every candidate. The authors build a real system (TLDChoiceNet v1 and v2), release a new dataset (TLDS), and propose two unsupervised metrics (DD and ACC). They are also transparent about limitations, which I appreciate.\n\nThe trouble is that the central claim is not supported by the experiments. The headline numbers tell the story: TLDChoiceNet v2 reaches test MSE 0.031, but the test split shares the fine-tuning dataset with training. That is not an independent test of generalization to a new fine-tuning dataset. The paper itself reports that v1's MSE jumps to 0.46 with predictions off by up to 80% when both the transfer learning and fine-tuning datasets are unseen. That alone undermines the claim that the method can choose a dataset 'given a fine-tuning dataset.'\n\nThe DD and ACC regressions are also in-sample fits. They take nine data points (three similar, six dissimilar plant/bird subsets), fit a line, and report R2 of 0.894 and 0.974. That's correlation, not prediction. A proper evaluation would hold out some TL datasets, or test on a different fine-tuning task. The paper lists multiple fine-tuning datasets as future work, which is honest but leaves the main claim unvalidated.\n\nWhat's genuinely new: the per-class embedding with ResNet50 v2, the LoRA-similar reduction, and the observation that pre-trained weights separate dissimilar classes more in latent space. Those are worth building on. The TLDS dataset, despite its narrow scope, is a useful artifact.\n\nSoft spots, in order: single fine-tuning task, shared test split, in-sample R2, and the admitted failure on unseen fine-tuning datasets. These are load-bearing, not cosmetic.\n\nWho is this for? Researchers working on transfer learning or dataset selection. They'll find the idea interesting, but they should not cite the quantitative results as evidence.\n\nFor peer review: send it out, but a good referee should reject it in current form. The evaluation needs at least one additional fine-tuning dataset and proper out-of-sample testing for the metrics. The authors seem capable of doing that.","headline":"Genuinely useful idea and honest writing, but the central claim is unvalidated: all headline results are in-sample or share the fine-tuning dataset, and the one out-of-sample number is bad.","tokens_in":8915,"tokens_out":3169,"would_cite":false,"duration_ms":31862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TLDChoiceNet predicts post-fine-tuning test accuracy for a candidate transfer-learning dataset, and the unsupervised class-correlation metric ACC reaches R2=0.97 when ranking candidate datasets.","keywords":["transfer learning","dataset selection","fine-tuning","accuracy prediction","distribution distance","average class correlation","ImageNet embeddings","image classification"],"falsifier":"Run the same 57-point TLDS-generation pipeline on multiple diverse fine-tuning datasets, such as medical images, satellite scenes, and general objects, and check whether ACC's linear fit to fine-tuned accuracy stays near R2 = 0.97 and TLDChoiceNet v2 keeps its held-out MSE near 0.031; a substantial drop in either number would refute the claimed generalization.","tokens_in":7885,"feed_emoji":"🎯","tokens_out":6873,"duration_ms":66947,"temperature":0.7,"pith_summary":"The paper sets out to turn transfer-learning dataset choice into a quantitative decision rather than a guess. It introduces TLDChoiceNet, a network that takes a fine-tuning dataset and a candidate transfer-learning dataset, along with its pre-trained model, and outputs the test accuracy that would result after fine-tuning. Trained on a newly built Transfer Learning Dataset spanning similar, dissimilar, random, and no-transfer cases, version 2 reaches 0.031 mean-squared error on held-out points from the same fine-tuning task. The paper also proposes two unsupervised metrics, distribution distance (DD) and average class correlation (ACC), which linearly explain fine-tuned accuracy with R2 of 0.89 and 0.97 respectively. A sympathetic reader would care because, if the method generalizes, practitioners can rank candidate pretraining datasets without paying the cost of fine-tuning each one.","feed_headline":"Predict which pretraining dataset wins before fine-tuning","feed_subtitle":"Training-free metric ACC reaches R2=0.97 against fine-tuned accuracy, ranking candidates without fine-tuning.","key_machinery":"The load-bearing machinery is a dataset-to-embedding-to-accuracy pipeline. On the fine-tuning side, each class is represented by the mean ImageNet-pretrained ResNet50 v2 activation over its images; on the candidate side, each class is represented by the mean last-convolutional-layer activation of the transfer-learned model over its images. Both are arranged as $10 \\times d$ matrices, reduced through learned linear layers into a shared 256-dimensional space, added together, and passed to a two-layer head that regresses to post-fine-tuning test accuracy. The paper's unsupervised alternative replaces the learned head with two closed-form metrics: DD, a Euclidean-style distance in mean, absolute skew, and absolute kurtosis of normalized pixel values, and ACC, the average pairwise correlation between per-class ImageNet embeddings of candidate and target datasets. The $R^2 = 0.974$ achieved by ACC is the number that carries the practical claim that a pretraining dataset can be chosen without training a predictor.","core_discovery":"TLDChoiceNet's central claim is that the outcome of transfer learning, namely the test-set accuracy after fine-tuning on a small target dataset, can be predicted from dataset statistics and model activations before fine-tuning is done. Version 1 achieves 0.154 MSE, while version 2, which replaces class-agnostic embeddings with per-class averaged activations from an ImageNet-pretrained ResNet50 v2 on the fine-tuning side and last-convolutional activations of the transfer-learned model on the candidate side, cuts this to 0.031 MSE. The paper further claims that the unsupervised metric ACC, defined as the mean pairwise correlation between per-class ResNet50 embeddings of the candidate and target datasets, regresses against fine-tuned accuracy with $R^2 = 0.974$, making it a cheap substitute for the learned predictor. The supporting observation is that pre-trained ImageNet features push dissimilar classes further apart in latent space, which is why ACC separates similar from dissimilar transfer datasets more sharply than raw pixel statistics.","pith_inferences":["Editorial inference: the same ACC recipe could be adapted to non-image domains by swapping in a per-class embedding from a language or audio encoder, though the paper only demonstrates image classification.","Editorial inference: because the reported test points share the fine-tuning dataset with training, the method's value for an arbitrary unseen target task is not yet established; the paper itself notes in Section 4.1 that v1's MSE degrades to 0.46 when both datasets are unseen.","Editorial inference: a parameter-free metric like ACC could be plugged into subset-selection loops to search over candidate class subsets or pretraining corpora without training a model per candidate.","Editorial inference: the paper's claim that low-level statistics explain transfer learning suggests that data curation for pretraining could target distributional similarity to downstream tasks rather than sheer scale."],"forward_implications":["A practitioner can rank candidate pretraining datasets by predicted fine-tune accuracy and fine-tune only the top choice, replacing multiple full training runs with one regression call.","The ACC metric provides a training-free ranking rule: embed each class with a pre-trained image encoder, average pairwise class correlations, and choose the candidate with the highest ACC.","Dataset low-level statistics and activation correlations carry enough transfer signal that no labels from the target task are needed to compare candidate transfer datasets.","Pre-trained embeddings that push dissimilar classes apart geometrically improve the separation between good and bad transfer datasets, so better encoders should yield sharper dataset-selection signals."],"supporting_citations":[{"why":"Supplies the ImageNet dataset and the pre-trained ResNet50 v2 weights used for the fine-tuning embedding and for the ACC metric.","marker":"[2]"},{"why":"Supplies the plant-disease dataset used as the similar transfer-learning dataset in the TLDS.","marker":"[7]"},{"why":"Supplies the plant-leaves fine-tuning dataset on which all accuracy labels are measured.","marker":"[1]"},{"why":"Supplies the bird-species dataset used as the dissimilar transfer-learning dataset.","marker":"[5]"},{"why":"Supplies CIFAR-100, a source of random transfer-learning dataset subsets in the TLDS.","marker":"[9]"},{"why":"Supplies the centered kernel alignment metric used to compare pre-trained and fine-tuned weights, supporting the paper's weight-similarity analysis.","marker":"[8]"},{"why":"Provides prior evidence that low-level data statistics drive transfer benefits, which the paper builds on and tests with its DD and ACC metrics.","marker":"[13]"}],"fun_headline_variants":["ACC metric ranks transfer datasets with R2=0.97","Pick transfer learning dataset without fine-tuning","Predict fine-tune accuracy using dataset correlations","TLDChoiceNet: choose transfer set via per-class embeddings","Training-free metric ACC beats learned predictor for transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's practical promise rests on assuming that the relationship learned on a single 22-class plant-leaf fine-tuning dataset holds for other target tasks; the paper's own hold-out results show that when both the transfer dataset and the fine-tuning dataset are unseen, version 1's prediction error rises to 0.46 MSE with errors up to 80 percent.","fun_headline_variants_meta":{"raw":{"variants":["ACC metric ranks transfer datasets with R2=0.97","Pick transfer learning dataset without fine-tuning","Predict fine-tune accuracy using dataset correlations","TLDChoiceNet: choose transfer set via per-class embeddings","Training-free metric ACC beats learned predictor for transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1422,"prompt_tokens":982,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":598,"tokens_out":440,"duration_ms":5684,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:40:15.157177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 57-point TLDS-generation pipeline on multiple diverse fine-tuning datasets, such as medical images, satellite scenes, and general objects, and check whether ACC's linear fit to fine-tuned accuracy stays near R2 = 0.97 and TLDChoiceNet v2 keeps its held-out MSE near 0.031; a substantial drop in either number would refute the claimed generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet dataset and the pre-trained ResNet50 v2 weights used for the fine-tuning embedding and for the ACC metric."},{"cited_title":"2009 IEEE conference on computer vision and pattern recognition , pages =","cited_arxiv_id":null,"evidence_quote":"Supplies the bird-species dataset used as the dissimilar transfer-learning dataset."},{"cited_title":"An online database for plant image analysis software tools , journal =","cited_arxiv_id":null,"evidence_quote":"Supplies the centered kernel alignment metric used to compare pre-trained and fine-tuned weights, supporting the paper's weight-similarity analysis."}],"review_version":1}