Pith. sign in

REVIEW 4 major objections 5 minor 17 references

TLDChoiceNet: Quantitatively Choosing a Transfer Learning Dataset

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TLDChoiceNet predicts post-fine-tuning test accuracy for a candidate transfer-learning dataset, and the unsupervised class-correlation metric ACC reaches R2=0.97 when ranking candidate datasets.

desk verdict Genuinely useful idea and honest writing, but the central claim is unvalidated: all headline results are in-sample or share the fine-tuning dataset, and the one out-of-sample number is bad. read the letter →

arxiv 2608.09091 v1 pith:ML3EIIDY submitted 2026-08-10 cs.CV cs.AI

Jing Ning , James D. Braza This is my paper · ORCID
classification cs.CVcs.AI
keywords transferlearningdatasetselectionfine-tuningaccuracypredictiondistributiondistanceaverageclasscorrelationImageNetembeddingsclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to turn transfer-learning dataset choice into a quantitative decision rather than a guess. It introduces TLDChoiceNet, a network that takes a fine-tuning dataset and a candidate transfer-learning dataset, along with its pre-trained model, and outputs the test accuracy that would result after fine-tuning. Trained on a newly built Transfer Learning Dataset spanning similar, dissimilar, random, and no-transfer cases, version 2 reaches 0.031 mean-squared error on held-out points from the same fine-tuning task. The paper also proposes two unsupervised metrics, distribution distance (DD) and average class correlation (ACC), which linearly explain fine-tuned accuracy with R2 of 0.89 and 0.97 respectively. A sympathetic reader would care because, if the method generalizes, practitioners can rank candidate pretraining datasets without paying the cost of fine-tuning each one.

What carries the argument

The load-bearing machinery is a dataset-to-embedding-to-accuracy pipeline. On the fine-tuning side, each class is represented by the mean ImageNet-pretrained ResNet50 v2 activation over its images; on the candidate side, each class is represented by the mean last-convolutional-layer activation of the transfer-learned model over its images. Both are arranged as $10 \times d$ matrices, reduced through learned linear layers into a shared 256-dimensional space, added together, and passed to a two-layer head that regresses to post-fine-tuning test accuracy. The paper's unsupervised alternative replaces the learned head with two closed-form metrics: DD, a Euclidean-style distance in mean, absolute skew, and absolute kurtosis of normalized pixel values, and ACC, the average pairwise correlation between per-class ImageNet embeddings of candidate and target datasets. The $R^2 = 0.974$ achieved by ACC is the number that carries the practical claim that a pretraining dataset can be chosen without training a predictor.

What would settle it

Run the same 57-point TLDS-generation pipeline on multiple diverse fine-tuning datasets, such as medical images, satellite scenes, and general objects, and check whether ACC's linear fit to fine-tuned accuracy stays near R2 = 0.97 and TLDChoiceNet v2 keeps its held-out MSE near 0.031; a substantial drop in either number would refute the claimed generalization.

Watch

Extended reading notes

Core claim

TLDChoiceNet's central claim is that the outcome of transfer learning, namely the test-set accuracy after fine-tuning on a small target dataset, can be predicted from dataset statistics and model activations before fine-tuning is done. Version 1 achieves 0.154 MSE, while version 2, which replaces class-agnostic embeddings with per-class averaged activations from an ImageNet-pretrained ResNet50 v2 on the fine-tuning side and last-convolutional activations of the transfer-learned model on the candidate side, cuts this to 0.031 MSE. The paper further claims that the unsupervised metric ACC, defined as the mean pairwise correlation between per-class ResNet50 embeddings of the candidate and target datasets, regresses against fine-tuned accuracy with $R^2 = 0.974$, making it a cheap substitute for the learned predictor. The supporting observation is that pre-trained ImageNet features push dissimilar classes further apart in latent space, which is why ACC separates similar from dissimilar transfer datasets more sharply than raw pixel statistics.

Load-bearing premise

The method's practical promise rests on assuming that the relationship learned on a single 22-class plant-leaf fine-tuning dataset holds for other target tasks; the paper's own hold-out results show that when both the transfer dataset and the fine-tuning dataset are unseen, version 1's prediction error rises to 0.46 MSE with errors up to 80 percent.

Editorial extensions

If this is right

  • A practitioner can rank candidate pretraining datasets by predicted fine-tune accuracy and fine-tune only the top choice, replacing multiple full training runs with one regression call.
  • The ACC metric provides a training-free ranking rule: embed each class with a pre-trained image encoder, average pairwise class correlations, and choose the candidate with the highest ACC.
  • Dataset low-level statistics and activation correlations carry enough transfer signal that no labels from the target task are needed to compare candidate transfer datasets.
  • Pre-trained embeddings that push dissimilar classes apart geometrically improve the separation between good and bad transfer datasets, so better encoders should yield sharper dataset-selection signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same ACC recipe could be adapted to non-image domains by swapping in a per-class embedding from a language or audio encoder, though the paper only demonstrates image classification.
  • Editorial inference: because the reported test points share the fine-tuning dataset with training, the method's value for an arbitrary unseen target task is not yet established; the paper itself notes in Section 4.1 that v1's MSE degrades to 0.46 when both datasets are unseen.
  • Editorial inference: a parameter-free metric like ACC could be plugged into subset-selection loops to search over candidate class subsets or pretraining corpora without training a model per candidate.
  • Editorial inference: the paper's claim that low-level statistics explain transfer learning suggests that data curation for pretraining could target distributional similarity to downstream tasks rather than sheer scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces TLDChoiceNet, a network that takes a fine-tuning dataset and a candidate transfer-learning dataset (or its transfer-learned model) and predicts the post-fine-tuning test-set accuracy. Version 1 uses a PCA-based embedding and reports 0.154 MSE; version 2 uses per-class ImageNet-pretrained ResNet50 v2 activations for the fine-tuning dataset and per-class TransferModel activations for the transfer dataset, reporting 0.031 MSE. The paper also proposes two unsupervised metrics, distribution distance (DD) and average class correlation (ACC), and reports linear-regression R^2 values of 0.894 and 0.974 against fine-tune accuracy. All experiments use one fine-tuning dataset (22-class plant leaves), and the paper explicitly states that the reported test split shares the fine-tuning dataset with training. When tested on an unseen fine-tuning dataset, TLDChoiceNet v1 degrades to MSE 0.46 with predictions off by up to 80%. The authors list testing with multiple fine-tuning datasets as future work.

Significance. If the central claim held, the method would give practitioners a way to select a pretraining dataset without fine-tuning every candidate, and the DD/ACC metrics would provide a cheap unsupervised proxy. The paper has useful ingredients: a clearly specified algorithm, a created dataset of transfer-learning tuples, and a public code link. However, the current empirical support is far too narrow for the stated claim. The headline numbers come from a single target task, the DD/ACC regressions are in-sample fits on nine points, and the one out-of-domain fine-tuning result reported is a large degradation. The paper is honest about these limitations, but the limitations directly undercut the abstract's promise of a general method.

major comments (4)
  1. [Section 4.1 and Section 5.3] The central claim that TLDChoiceNet chooses the best transfer learning dataset 'given a fine tuning dataset' is not supported because all reported TLDChoiceNet results use the same 22-class plant-leaves fine-tuning dataset. The only result with an unseen fine-tuning dataset, reported in Section 4.1, is v1's MSE of 0.46 with predictions off by up to 80%; no analogous result is given for v2, DD, or ACC. Section 5.3 explicitly lists testing with multiple fine-tuning datasets as future work, so the current evidence does not establish the claimed generality.
  2. [Section 4.2, Tables 2 and 3] The DD and ACC R^2 values of 0.894 and 0.974 are computed from linear regressions fit to the same nine TLDS points that are used to report R^2. These are in-sample goodness-of-fit values, not predictive validations. With nine points split into a similar cluster and a dissimilar cluster, a two-group linear fit can produce a high R^2 without demonstrating that the metric discriminates among datasets within either group. The paper needs held-out validation or cross-validation, and ideally multiple target tasks, before claiming that DD or ACC can 'choose' a transfer dataset.
  3. [Section 3.2 and Section 4.2] The DD and ACC regressions ignore most of the TLDS. Tables 2 and 3 list only similar (three rows) and dissimilar (six rows) transfer-learning datasets, whereas Section 3.2 states that the TLDS also includes 40 random subsets from CIFAR-100 and ImageNet and a no-transfer condition. The random subsets are exactly the kind of candidates a general dataset-choice method must handle, so the reported correlations do not cover the claimed scope of the method. The paper should either include these cases in the regressions or justify their exclusion.
  4. [Section 4.2 and Tables 2/3] The paper does not compare DD or ACC against simple baselines, such as predicting accuracy from the transfer dataset's source category, dataset size, or the no-transfer accuracy. Given that the reported tables contain only two broad categories (plant diseases and bird species), a trivial rule that assigns a high accuracy to similar datasets and a low accuracy to dissimilar datasets could reproduce much of the observed separation. A quantitative comparison to such baselines is necessary to support the claim that low-level statistics and class correlations are the explanatory factors.
minor comments (5)
  1. [Section 4.2] The text states that the average ACC for dissimilar TL datasets is 0.276, but the six values in Table 3 average to approximately 0.271; the discrepancy should be corrected or the averaging procedure clarified.
  2. [References] References [9] and [10] are duplicate entries for the same Krizhevsky technical report; one of them should be removed or differentiated.
  3. [Section 4.1 and Figure 5] The paper reports training on 94 datapoints and shows test predictions in Figure 5, but it never states the number of test datapoints or the total TLDS size, which makes the MSE values difficult to interpret statistically. The test set size should be reported in the text.
  4. [Section 3.2] Section 3.2 says the similar and dissimilar TL datasets each have three randomly sampled subsets, but Tables 2 and 3 list three similar rows and six dissimilar rows; this inconsistency should be resolved.
  5. [Throughout] The phrase 'MSE loss' is redundant; 'MSE' is sufficient.

Circularity Check

1 steps flagged · score 5.0 of 10

The DD/ACC headline R2 values are in-sample regression fits presented as predictive evidence, while TLDChoiceNet's own test is confined to a single fine-tuning dataset.

  1. fitted input called prediction [Abstract, Sections 3.4.1-3.4.2, 4.2, and Conclusion (Section 5)]
    "Distribution distance (DD), which linearly regresses against fine-tune accuracy with an R2 of 0.89, and average class correlation (ACC), which improves the R2 to 0.97. ... We then calculate the DD metric for each transfer learning dataset and linearly regress it with test-set accuracy to analyze the correlation. ... After fitting DD with the test-set accuracy, it can explain the variation in accuracy with a R2 of 0.894 ... after fitting the correlation metric ACC with the test fine-tune accuracy, it can explain the variation in fine-tune accuracy with a R2 of 0.974."

    The same nine (metric, accuracy) points are used both to estimate the linear regression coefficients and to compute the reported R2, so each R2 is an in-sample goodness-of-fit rather than a validated forecast. The paper nevertheless presents these fits as evidence that DD and ACC 'can determine the optimal choice of transfer learning dataset' and can serve as a 'high-quality substitute for test-set accuracy.' With n=9 and a single fine-tuning dataset, the fit mostly separates the two clusters of similar versus dissimilar transfer datasets; no held-out or cross-validated calibration is reported. The advertised predictive/choice ability of the metrics therefore reduces to a post-hoc fit to the very accuracy labels it is supposed to predict.

full rationale

No load-bearing self-citation or imported-uniqueness chain is present; the references are external prior work, and the author-contribution statements are not used as evidence. TLDChoiceNet's MSE numbers are not circular by construction: the network is trained on TLDS tuples and evaluated on held-out TLDS tuples, and its embeddings are not derived from the accuracy labels. The limitation there is generalization, not circularity: the held-out tuples all share the same fine-tuning dataset, and Section 4.1 admits that with an unseen fine-tuning dataset v1's MSE rises to 0.46 with predictions off by up to 80%, with no equivalent unseen-fine-tuning-dataset evaluation reported for v2 or for DD/ACC. The clear circular step is the unsupervised-metric result: the R2 values are computed on the same data used to fit the regressions, so the 'optimal choice' claim is supported by in-sample calibration rather than independent prediction. This is partial circularity, not a full derivation-by-definition.

Assumptions & free parameters 6 free parameters · 4 assumptions · 3 invented entities

The central claims rest on several unvalidated assumptions: one fine-tuning task, ImageNet embeddings as a universal similarity space, and pixel-level moments capturing transfer-relevant structure. The DD/ACC correlations are fit in-sample on nine datasets, so the reported R2 values are not predictions.

free parameters (6)
  • fine-tuning image count = 1000 per fine-tuning dataset
    The paper states that 1000 fine-tuning images strikes a reasonable balance (Section 3.3.1); this choice affects all accuracy measurements.
  • number of classes retained in fine-tuning embedding (ChoiceNet v2) = 10
    Algorithm 1 keeps the 10 classes with largest L1 norm; no ablation is provided for this choice.
  • LoRA-similar reduction dimension = 256
    The learned projection size is set to 256 in both branches (Algorithm 1); no sensitivity analysis is given.
  • DD linear regression coefficients = not reported
    The R2=0.894 is from a linear fit of DD to fine-tune accuracy on the same 9 data points; coefficients are not reported or validated out-of-sample.
  • ACC linear regression coefficients = not reported
    The R2=0.974 is from a linear fit of ACC to fine-tune accuracy on the same 9 data points; coefficients are not reported or validated out-of-sample.
  • Training and fine-tuning epochs and learning rates = 15 epochs, lr 0.001 for pretraining; 15 epochs, lr 0.0001 for fine-tuning
    These hyperparameters define TLDS accuracy labels but no rationale or ablation is given.
assumptions (4)
  • domain assumption ImageNet-pretrained ResNet50 v2 activations are a valid similarity space for arbitrary image datasets.
    Used to embed fine-tuning and transfer datasets in ChoiceNet v2 and ACC (Sections 3.3.3, 3.4.2).
  • domain assumption The TransferModel CNN with frozen first two Conv2 blocks is representative of transfer learning behavior.
    All TLDS accuracy labels come from this architecture (Section 3.2).
  • domain assumption Pixel-level mean, skew, and kurtosis capture dataset distribution shift relevant to transfer.
    The DD metric is built from these moments (Section 3.4.1).
  • domain assumption A single fine-tuning run per dataset is a stable estimate of transfer performance.
    No repeated runs or error bars are reported for TLDS accuracy values.
invented entities (3)
  • Transfer Learning Dataset (TLDS)
    purpose: A new dataset of (transfer dataset, model, accuracy) triples used to train and evaluate TLDChoiceNet.
    The dataset is generated from external Kaggle/ImageNet/CIFAR sources but is not released; no external validation of its representativeness is provided.
  • Distribution Distance (DD)
    purpose: An unsupervised metric combining mean, skew, and kurtosis differences between pixel distributions.
    Proposed in Section 3.4.1; correlation with accuracy is evaluated only in-sample on 9 TL datasets.
  • Average Class Correlation (ACC)
    purpose: An unsupervised metric averaging correlations between per-class ImageNet embeddings of two datasets.
    Proposed in Section 3.4.2; correlation with accuracy is evaluated only in-sample on 9 TL datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TLDChoiceNet: Quantitatively Choosing a Transfer Learning Dataset." pith.science (2026). https://pith.science/paper/ML3EIIDY

@misc{pith2026260809091,
  author       = {Pith},
  title        = {Pith review of: TLDChoiceNet: Quantitatively Choosing a Transfer Learning Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ML3EIIDY}},
  note         = {Machine review of arXiv:2608.09091}
}
read the original abstract

Transfer learning is particularly useful in settings with limited training data, and within image classification it is common to transfer learn upon massive datasets like ImageNet , CIFAR-100, or COCO . Qualitatively, it seems a transfer learning dataset should have both more classes and more examples per class than the fine tuning dataset; however, a quantitative method to choose the best transfer learning dataset does not currently exist. In this paper, we design TLDChoiceNet, a model to choose the best transfer learning dataset given a fine tuning dataset by predicting the test-set accuracy after fine-tuning. A simple version 1 achieves 0.154 MSE on the test dataset, while a version 2 leveraging an ImageNet pre-trained ResNet50 v2 embedding with per-class information attains a 5X lower MSE of 0.031. We further design two metrics that enable an unsupervised method of choosing an optimal transfer learning dataset: distribution distance (DD), which linearly regresses against fine-tune accuracy with an R2 of 0.89, and average class correlation (ACC), which improves the R2 to 0.97. Our results underscore that a dataset's low-level statistics can explain the transfer learning effect, and that using a pre-trained ImageNet can embed different classes further apart in latent feature space.

Figures

Figures reproduced from arXiv: 2608.09091 by the authors.

Figure 1
Figure 1. Overview of TLDChoiceNet. Given a target fine-tuning dataset and one or more candidate transfer learning datasets (and/or their pre-trained models), TLDChoiceNet embeds each input, reduces it through a learned LoRA-similar layer, and predicts the post-fine-tuning test-set accuracy. This lets a practitioner choose the best transfer learning dataset without actually fine-tuning on every candidate. • We introduce the T… view at source ↗
Figure 2
Figure 2. The Transfer Learning Dataset (TLDS) was created with a custom CNN (“TransferModel”) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Each version of TLDChoiceNet builds upon the previous. All figures show the embedding [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Dataset Embedding via ResNet50 v2 Now, we can define a new metric: average class correlation (ACC). We have dataset i with n classes and dataset j of m classes. Let Corkl denote the correlation between the per-class embedding of class k in dataset i and class l in data…
Figure 5
Figure 5. Figure 5: TLDChoiceNet v1 attained a MSE loss of 0.154 and v2 attained a MSE loss of 0.031 on [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Histogram of Normalized Pixel Values. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Distance Metric DD’s Fitting Result. A.2 Activation Map Correlations A.3 Pre-Trained Weights Across Fine-Tuning B Source Code All source code for the project can be found on GitHub at CDC1688/TLDChoiceNet. TL Dataset Correlation Metric (ACC) Fine-tune Accuracy Similar …
Figure 8
Figure 8. Figure 8: Correlation Metric ACC vs Test-Set Accuracy. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Correlation Embedding Metric ACC. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Weight change and CKA. cases, re-runnable to generate more datapoints per invocation. J.D.B. designed the ChoiceNet v2 embeddings and architecture, building upon the ChoiceNet v1 work, and created scripts to connect ChoiceNet v1 and v2 training with the TLDS. 15 [PIT…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    Meta-Learning for Few-Shot Land Cover Classification , journal =

    Marc Ru. Meta-Learning for Few-Shot Land Cover Classification , journal =. 2020 , url =. 2004.13390 , timestamp =

    arXiv 2020

  2. [2]

    Plant Leaves for Image Classification , howpublished =

    work page

  3. [3]

    Hughes and Marcel Salath

    David P. Hughes and Marcel Salath. An open access repository of images on plant health to enable the development of mobile disease diagnostics through machine learning and crowdsourcing , journal =. 2015 , url =

    work page 2015

  4. [4]

    BIRDS 450 SPECIES- IMAGE CLASSIFICATION , howpublished =

    work page

  5. [5]

    2009 IEEE conference on computer vision and pattern recognition , pages =

    Imagenet: A large-scale hierarchical image database , author =. 2009 IEEE conference on computer vision and pattern recognition , pages =. 2009 , organization =

    work page 2009

  6. [6]

    2009 , type =

    Alex Krizhevsky , title =. 2009 , type =

    work page 2009

  7. [7]

    Microsoft

    Tsung. Microsoft. CoRR , volume =. 2014 , url =. 1405.0312 , timestamp =

    arXiv 2014

  8. [8]

    An online database for plant image analysis software tools , journal =

    Lobet, Guillaume and Draye, Xavier and P. An online database for plant image analysis software tools , journal =. 2013 , month =. doi:10.1186/1746-4811-9-38 , url =

    work page doi:10.1186/1746-4811-9-38 2013

Show all 17 references
  1. [9]

    Alina Kuznetsova and Hassan Rom and Neil Alldrin and Jasper R. R. Uijlings and Ivan Krasin and Jordi Pont. The Open Images Dataset. CoRR , volume =. 2018 , url =. 1811.00982 , timestamp =

    2018 arXiv

  2. [10]

    Learning multiple layers of features from tiny images , year =

    Krizhevsky, Alex and Hinton, Geoffrey , address =. Learning multiple layers of features from tiny images , year =

  3. [11]

    CoRR , volume =

    Dan Hendrycks and Kimin Lee and Mantas Mazeika , title =. CoRR , volume =. 2019 , url =. 1901.09960 , timestamp =

    2019 arXiv

  4. [12]

    CoRR , volume =

    Weifeng Ge and Yizhou Yu , title =. CoRR , volume =. 2017 , url =. 1702.08690 , timestamp =

    2017 arXiv

  5. [13]

    Peters and Sebastian Ruder and Noah A

    Matthew E. Peters and Sebastian Ruder and Noah A. Smith , title =. CoRR , volume =. 2019 , url =. 1903.05987 , timestamp =

    2019 arXiv

  6. [14]

    Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution , publisher =

    Kumar, Ananya and Raghunathan, Aditi and Jones, Robbie and Ma, Tengyu and Liang, Percy , keywords =. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution , publisher =. 2022 , copyright =. doi:10.48550/ARXIV.2202.10054 , url =

    2022 doi arXiv

  7. [15]

    CoRR , volume =

    Christopher Fifty and Ehsan Amid and Zhe Zhao and Tianhe Yu and Rohan Anil and Chelsea Finn , title =. CoRR , volume =. 2021 , url =. 2109.04617 , timestamp =

    2021 arXiv

  8. [16]

    CoRR , volume =

    Behnam Neyshabur and Hanie Sedghi and Chiyuan Zhang , title =. CoRR , volume =. 2020 , url =. 2008.11687 , timestamp =

    2020 arXiv

  9. [17]

    Hinton , title =

    Simon Kornblith and Mohammad Norouzi and Honglak Lee and Geoffrey E. Hinton , title =. CoRR , volume =. 2019 , url =. 1905.00414 , timestamp =

    2019 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.