Pith. sign in

REVIEW 4 major objections 5 minor 69 references

MetaRank: Task-Aware Metric Selection for Model Transferability Estimation

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A short text description of a target dataset predicts which model-transferability metric will rank pretrained models best, and a meta-learned ranker using it beats every fixed metric on eleven datasets.

desk verdict A worthwhile idea with a clear evaluation gap: MetaRank's gains over SFDA are too small to trust without significance tests, and the abstract does not describe the method in the paper. read the letter →

arxiv 2511.21007 v2 pith:HF4LUGCJ submitted 2025-11-26 cs.CV

Yuhang Liu , Wenjie Zhao , Xin Wang , Yunhui Guo This is my paper
classification cs.CV
keywords transferabilityestimationmodelselectionmeta-learninglearningtoranktextembeddingsdatasetmeta-featurestransferpretrainedmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that no single transferability metric is best for every dataset and that the right metric can be chosen ahead of time from a short textual description of the target dataset. It introduces MetaRank, which encodes dataset and metric descriptions into a shared semantic space and trains a ranker with a listwise objective so that, for an unseen dataset, it orders candidate metrics and picks the top one. On eleven image datasets and nine transferability metrics, MetaRank reports the lowest average and median rank among fixed metrics and other meta-learners, with most per-dataset ranks in the top five.

What carries the argument

The central object is a learned mapping from a (dataset description, metric description) pair to a relevance score, trained to reproduce the rank order of metrics observed on past datasets. Structured text descriptions are embedded by a fixed pretrained language model, so new datasets and new metrics enter the framework simply by writing a description. The ranking objective is listwise NDCG, which discounts lower-ranked positions and therefore prioritizes identifying the top-1 or top-k metric. At test time only the relative order of the predicted scores matters, so the framework never needs to compute absolute transferability values.

What would settle it

Hold out a dataset whose description is deliberately misleading, e.g., describing a fine-grained dataset like Cars with coarse wording, and check whether MetaRank still selects the metric that is empirically best on the true dataset. If the chosen metric's weighted-Kendall rank drops well below a strong fixed metric under such a mismatch, the text schema, not the underlying task, is driving the selection.

Watch

Extended reading notes

Core claim

Model transferability estimation (MTE) produces cheap scores that rank pretrained source models for a target dataset without fine-tuning. MetaRank's central claim is that the quality of these metrics is task-dependent and that the best metric for a new dataset can be selected in advance by learning from historical dataset-metric rankings. The selection is formulated as a learning-to-rank problem: a pretrained language model embeds structured textual descriptions of the dataset (image count, class count, content summary, label granularity) and of each metric into a shared semantic space; a tree-boosting meta-predictor scores each dataset-metric pair; and a listwise NDCG objective emphasizes g

Load-bearing premise

The load-bearing premise is that a short, hand-written textual description of a target dataset—image count, number of classes, content summary, label granularity—contains enough information to determine which MTE metric will rank pretrained models best, and that this relationship learned on ten training datasets transfers to an unseen dataset.

Editorial extensions

If this is right

  • On an unseen dataset, the only new computation is embedding its description; precomputed metric embeddings and the trained predictor are reused, keeping the selection cheap.
  • A new MTE metric can be incorporated with one textual description and one encoding step, after which it can be scored against any dataset without recomputing all historical metrics.
  • Because the decision is ordinal, the predictor does not need to estimate actual transferability values, only their order, which is a simpler and more robust target than regression.
  • The zero-shot experiment—training on six metrics and ranking nine, including three unseen—indicates that the learned notion of metric similarity transfers to metrics never observed in training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates the selected metric by its weighted-Kendall rank, not by the downstream accuracy of the model the metric would pick; a direct practical test would chain MetaRank into full fine-tuning and measure whether the chosen source model actually transfers better.
  • The same text-embedding-plus-listwise-ranker recipe could apply to other method-selection problems, such as out-of-distribution detection, optimizer choice, or augmentation policy, wherever both the query and the candidates can be described by short text.
  • The component ablation suggests label granularity carries the most signal; a natural stress test is to feed the ranker deliberately incomplete descriptions across many more datasets and measure how far performance degrades beyond the single-component removals reported.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MetaRank, a meta-learning framework for selecting a model transferability estimation (MTE) metric for an unseen target dataset. It encodes textual descriptions of datasets and MTE metrics with a sentence transformer, concatenates the embeddings, and trains an XGBoost learning-to-rank model with an NDCG objective. On a benchmark of 11 ImageNet-pretrained models, 9 MTE metrics, and 11 target datasets, evaluated via leave-one-dataset-out, the authors report that MetaRank achieves the lowest average rank among fixed-metric and meta-learner baselines.

Significance. The paper tackles a relevant and under-addressed problem: MTE metrics vary in effectiveness across target datasets, and selecting the right metric automatically is valuable. The use of language-model embeddings of dataset and metric descriptions as meta-features is a simple, computationally cheap, and potentially generalizable idea, and the LODO evaluation over 11 datasets is a reasonable first benchmark. However, the empirical evidence for the headline claim is not yet conclusive: the reported advantage over the strongest fixed metric (SFDA) is small and unsupported by any statistical test, and the abstract's description of the method does not match the actual implementation. If the statistical concerns are addressed, the work could make a useful contribution.

major comments (4)
  1. [Abstract vs. Section 4] The abstract describes MetaRank as a retrieve-and-rerank cascade that represents metrics via pairwise source-model ordering and uses a product-kernel regressor. Section 4 presents neither of these components: the actual method is a single-stage XGBoost ranker over concatenated sentence embeddings, optimized with a listwise NDCG objective. This is a fundamental mismatch between the claimed and implemented method. Please either implement the described architecture or revise the abstract to accurately reflect the proposed method.
  2. [Section 5.2, Table 6] The central claim that MetaRank 'significantly outperforms all compared baselines' is not supported by inferential statistics. The average-rank difference between MetaRank (4.77) and SFDA (5.18) over 11 datasets is 0.41 rank units, and no significance test, confidence interval, or variance estimate is provided. Moreover, the Random baseline is a single draw per dataset, so its rank (8.45) has no sampling distribution. A paired permutation test or Wilcoxon signed-rank test across the 11 per-dataset ranks, and/or bootstrapped intervals, is required before claiming statistical superiority.
  3. [Section 4.1, Table 3, Fig. 10] The dataset textual descriptions are handcrafted by the authors and explicitly include label granularity, which the ablation in Fig. 10 finds to be the most influential component. Because the authors wrote these descriptions with knowledge of the downstream results, the descriptions may inadvertently encode the experimental outcome (e.g., labeling a dataset as 'fine-grained' is correlated with SFDA's strong performance). This creates a risk that the LODO evaluation overestimates generalization to datasets where such expert-crafted descriptions are not available. Please analyze the sensitivity to description wording or discuss how descriptions can be obtained in practice without this potential leak.
  4. [Section 5.3, Fig. 6] The zero-shot generalization experiment, which tests on three metrics not seen during training, shows high variance over random seeds: some runs perform no better than the baselines. This variability is acknowledged in the text, yet the section concludes with 'strong evidence of successful generalization.' The authors should temper this conclusion and report multiple-seed results with error bars for the main evaluation as well, since the stability of the method is not established.
minor comments (5)
  1. [Abstract] The abstract states the experiments use 13 candidate MTE metrics and 14 target datasets, but Section 5.1 uses 9 metrics and 11 datasets. Please correct the numbers.
  2. [Section 5.1] With 11 datasets, leave-one-dataset-out leaves 10 datasets for training/validation, not 'nine' as written. Also, clarify how the three validation datasets are chosen in each fold.
  3. [Fig. 5] The Random baseline is shown with a single point per dataset. Since the whole point of random selection is variability, please show multiple random seeds (e.g., boxplot of many draws) to make the comparison fair.
  4. [References] The citation of the no-free-lunch theorem [59] for the statement that no single MTE metric is universally optimal is imprecise; the theorem concerns optimization algorithms, not metrics. A more appropriate citation or a direct argument would be clearer.
  5. [Section 5.3] The conventional meta-feature ablation excludes SUN397 and Cars due to feature extraction failures; this should be stated in the main text, as it reduces the comparison to 9 datasets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MetaRank's LODO evaluation separates training targets from test predictions.

full rationale

MetaRank's central derivation is not circular by construction. The offline stage fits f_theta to predict ground-truth metric rankings r_j = argsort_k(-tau_w;j,k) from concatenated text embeddings [d_j; s_k] (Eq. 2), and the online stage scores a held-out dataset from its textual description only. Section 5.1 states 'We use a Leave-One-Dataset-Out (LODO) evaluation: in each fold, one target dataset is held out for testing while the remaining nine are used for training and validation,' so the target tau_w values are not inputs to the fitted model. The handcrafted dataset descriptions (image count, class count, content, label granularity) are features, not the prediction target; even though label granularity is identified in Sec. 3.2 as a driver of metric effectiveness, that is a legitimate feature choice rather than a self-definitional reduction. The ablation removing granularity tests this empirically. The only author self-citation is [19] (KITE), cited in the Related Work list of MTE metrics; it is not load-bearing for the central claim. The absence of significance testing (e.g., MetaRank average rank 4.77 vs. SFDA 5.18 on 11 datasets) is a statistical robustness concern, not a circularity. Hence no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

MetaRank does not introduce new physical or mathematical entities. Its free parameters are the XGBoost hyperparameters (unreported) and the handcrafted textual descriptions. The load-bearing axioms are empirical: text descriptions encode enough information to select a good MTE metric, and the 11-dataset benchmark is representative.

free parameters (2)
  • XGBoost hyperparameters
    Tuned by grid search on three validation datasets (Section 5.1), but the grid and chosen values are not reported; they influence ranking quality and are fit to the meta-training data.
  • Handcrafted dataset and metric textual descriptions = handcrafted strings
    The descriptions in Tables 2 and 3 are manually written to include properties the authors believe matter (e.g., label granularity). The ablation shows that removing granularity hurts most, indicating the descriptions are tuned to the observed phenomenon.
assumptions (3)
  • ad hoc to paper Textual description of a dataset is sufficient meta-feature to predict which MTE metric will rank models best.
    This is the core modeling assumption (Section 4.1, 4.2) and is not independently validated beyond the 11-dataset benchmark.
  • domain assumption The 11 target datasets and 11 ImageNet-pretrained models are a representative sample of transfer learning tasks.
    The evaluation and conclusions rely on the benchmark from [47]; no evidence is given that it spans the space of real-world transfer scenarios.
  • domain assumption The relationship between dataset properties and MTE metric effectiveness is stable across datasets.
    MetaRank's transferability rests on this meta-learning premise stated in Section 4: 'a metric that performs well on a new dataset is expected to have consistently excelled on historically similar ones.'

how reviews work

0 comments
Cite this review

Pith. "Pith review of MetaRank: Task-Aware Metric Selection for Model Transferability Estimation." pith.science (2026). https://pith.science/paper/HF4LUGCJ

@misc{pith2026251121007,
  author       = {Pith},
  title        = {Pith review of: MetaRank: Task-Aware Metric Selection for Model Transferability Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HF4LUGCJ}},
  note         = {Machine review of arXiv:2511.21007}
}
read the original abstract

Selecting an appropriate pre-trained source model is a critical, yet computationally expensive, task in transfer learning. Model Transferability Estimation (MTE) methods address this by providing efficient proxy metrics to rank models without full fine-tuning. In practice, the choice of which MTE metric to use is often ad hoc or guided simply by a metric's average historical performance. However, we observe that the effectiveness of MTE metrics is highly task-dependent and no single metric is universally optimal across all target datasets. To address this gap, we introduce MetaRank, a meta-learning framework for automatic, task-aware MTE metric selection. MetaRank adopts a retrieve-and-rerank cascade. A lightweight retrieval stage first narrows the metric pool using performance observed on similar meta-training datasets. To address the heterogeneous definitions and scales of MTE scores, the reranking stage represents each retrieved metric through the pairwise source-model ordering it induces on the target dataset. A product-kernel regressor combines dataset and ordering similarities to refine the ranking, enabling cross-metric transfer from historical metrics with similar ordering behavior. MetaRank then ranks the retrieved metrics and selects the most appropriate one to guide source-model selection on an unseen target dataset. Extensive experiments across 11 pre-trained models, 13 candidate MTE metrics, and 14 target datasets demonstrate that MetaRank significantly outperforms all compared baselines.

Figures

Figures reproduced from arXiv: 2511.21007 by the authors.

Figure 1
Figure 1. Comparison of ad-hoc vs. task-aware metric selec￾tion. Ad-hoc selection employs a single MTE metric (e.g., NCTI), which may achieve the highest average performance but fails to identify the optimal metric for each target dataset. In contrast, task-aware selection adapts the metric choice per dataset (e.g., SFDA for Aircraft and H-Score for Cars), enabling per-dataset op￾timal metric identification and improved trans… view at source ↗
Figure 2
Figure 2. Performance evaluation of MTE metrics across diverse [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed MetaRank for MTE metric selection. Offline training learns a meta-predictor [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Average Rank of Different Selection Methods on All Datasets. Lower rank, positioned to the left, indicates better performance. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Rank Distribution of Different Selection Methods. The orange line inside each box is the median; the box edges are the first [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Trained on a random subset of 6 metrics and tested on [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: MetaRank using tree-based XGBoost outperforms the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 10
Figure 10. Figure 10: Importance of textual components. Removing content [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 8
Figure 8. Figure 8: Comparison of different ranking objectives. MetaRank [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 11
Figure 11. Figure 11: Comparison of running time between MetaRank and [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 10 linked inside Pith

  1. [1]

    One size does not fit all in evaluating model selection scores for image clas- sification.Scientific Reports, 14(1):30239, 2024

    Nermeen Abou Baker and Uwe Handmann. One size does not fit all in evaluating model selection scores for image clas- sification.Scientific Reports, 14(1):30239, 2024. 4, 8

    2024

  2. [2]

    Task2vec: Task embedding for meta-learning

    Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Ste- fano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 6430–6439,

  3. [3]

    Edesio Alcobac ¸a, Felipe Siqueira, Adriano Rivolli, Lu´ıs P. F. Garcia, Jefferson T. Oliva, and Andr´e C. P. L. F. de Carvalho. Mfe: Towards reproducible meta-feature extraction.Journal of Machine Learning Research, 21(111):1–5, 2020. 2, 4, 8, 1

    2020

  4. [4]

    An information- theoretic approach to transferability in task transfer learning

    Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information- theoretic approach to transferability in task transfer learning. In2019 IEEE international conference on image processing (ICIP), pages 2309–2313. IEEE, 2019. 2, 3, 6, 4

    2019

  5. [5]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. InEuropean conference on computer vision, pages 446–461. Springer, 2014. 6, 3

    2014

  6. [6]

    Learning to rank using gradient descent

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. InProceedings of the 22nd in- ternational conference on Machine learning, pages 89–96,

  7. [7]

    Learning to rank: from pairwise approach to listwise ap- proach

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise ap- proach. InProceedings of the 24th international conference on Machine learning, pages 129–136, 2007. 3

    2007

  8. [8]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016. 6

    2016

Show all 69 references
  1. [9]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 6, 3

    2014

  2. [10]

    Which model to transfer? a survey on transferabil- ity estimation.arXiv preprint arXiv:2402.15231, 2024

    Yuhe Ding, Bo Jiang, Aijing Yu, Aihua Zheng, and Jian Liang. Which model to transfer? a survey on transferabil- ity estimation.arXiv preprint arXiv:2402.15231, 2024. 1, 2

    2024 arXiv

  3. [11]

    Pre-trained model reusability evaluation for small-data trans- fer learning.Advances in Neural Information Processing Systems, 35:37389–37400, 2022

    Yao-Xiang Ding, Xi-Zhu Wu, Kun Zhou, and Zhi-Hua Zhou. Pre-trained model reusability evaluation for small-data trans- fer learning.Advances in Neural Information Processing Systems, 35:37389–37400, 2022. 3

    2022

  4. [12]

    Automl using metadata language embeddings.arXiv preprint arXiv:1910.03698, 2019

    Iddo Drori, Lu Liu, Yi Nian, Sharath C Koorathota, Jie S Li, Antonio Khalil Moretti, Juliana Freire, and Madeleine Udell. Automl using metadata language embeddings.arXiv preprint arXiv:1910.03698, 2019. 3, 4

    1910 arXiv

  5. [13]

    The pascal visual object classes (voc) challenge.International journal of computer vision, 88(2):303–338, 2010

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge.International journal of computer vision, 88(2):303–338, 2010. 6, 3

    2010

  6. [14]

    Comparing top k lists.SIAM Journal on discrete mathemat- ics, 17(1):134–160, 2003

    Ronald Fagin, Ravi Kumar, and Dakshinamurthi Sivakumar. Comparing top k lists.SIAM Journal on discrete mathemat- ics, 17(1):134–160, 2003. 3

    2003

  7. [15]

    Large language models (llms) on tabular data: Prediction, generation, and understanding–a survey.arXiv preprint arXiv:2402.17944,

    Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srini- vasan Sengamedu, and Christos Faloutsos. Large language models (llms) on tabular data: Prediction, generation, and understanding–a survey.arXiv preprint arXiv:2402.17944,

    arXiv

  8. [16]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

    Li Fei-Fei. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004. 6, 3

    2004

  9. [17]

    Rankme: Assessing the downstream perfor- mance of pretrained self-supervised representations by their rank

    Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream perfor- mance of pretrained self-supervised representations by their rank. InInternational conference on machine learning, pages 10929–10974. PMLR, 2023. 2

    2023

  10. [18]

    Etran: Energy-based transferability estimation

    Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etran: Energy-based transferability estimation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 18613– 18622, 2023. 2, 6, 4

    2023

  11. [19]

    Kite: A kernel-based improved transferability estimation method.arXiv preprint arXiv:2405.01603, 2024

    Yunhui Guo. Kite: A kernel-based improved transferability estimation method.arXiv preprint arXiv:2405.01603, 2024. 1

    2024 arXiv

  12. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6

    2016

  13. [21]

    Neural collaborative filtering

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pages 173–182, 2017. 6, 4

    2017

  14. [22]

    Sup- port vector learning for ordinal regression

    Ralf Herbrich, Thore Graepel, and Klaus Obermayer. Sup- port vector learning for ordinal regression. In1999 Ninth International Conference on Artificial Neural Networks ICANN 99.(Conf. Publ. No. 470), pages 97–102. IET, 1999. 3

    1999

  15. [23]

    Meta-learning in neural networks: A survey

    Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelli- gence, 44(9):5149–5169, 2021. 2

    2021

  16. [24]

    Strate- gies for pre-training graph neural networks.arXiv preprint arXiv:1905.12265, 2019

    Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strate- gies for pre-training graph neural networks.arXiv preprint arXiv:1905.12265, 2019. 1

    1905 arXiv

  17. [25]

    Densely connected convolutional net- works

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kil- ian Q Weinberger. Densely connected convolutional net- works. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017. 6

    2017

  18. [26]

    Cumulated gain- based evaluation of ir techniques.ACM Transactions on In- formation Systems (TOIS), 20(4):422–446, 2002

    Kalervo J ¨arvelin and Jaana Kek ¨al¨ainen. Cumulated gain- based evaluation of ir techniques.ACM Transactions on In- formation Systems (TOIS), 20(4):422–446, 2002. 5, 6

    2002

  19. [27]

    Dataset2vec: Learning dataset meta-features.Data Mining and Knowledge Discovery, 35(3):964–985, 2021

    Hadi S Jomaa, Lars Schmidt-Thieme, and Josif Grabocka. Dataset2vec: Learning dataset meta-features.Data Mining and Knowledge Discovery, 35(3):964–985, 2021. 2 9

    2021

  20. [28]

    Isac–instance-specific algorithm configura- tion

    Serdar Kadioglu, Yuri Malitsky, Meinolf Sellmann, and Kevin Tierney. Isac–instance-specific algorithm configura- tion. InECAI 2010, pages 751–756. Ios Press, 2010. 6, 4

    2010

  21. [29]

    Do better imagenet models transfer better? InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661–2671, 2019

    Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661–2671, 2019. 1

    2019

  22. [30]

    Collecting a large-scale dataset of fine-grained cars

    Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. 2013. 2, 3, 6, 8

    2013

  23. [31]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 3, 4, 6

    2009

  24. [32]

    Ranking neural checkpoints

    Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2663– 2673, 2021. 2, 4, 6

    2021

  25. [33]

    Learning to rank for information re- trieval.Foundations and Trends® in Information Retrieval, 3(3):225–331, 2009

    Tie-Yan Liu et al. Learning to rank for information re- trieval.Foundations and Trends® in Information Retrieval, 3(3):225–331, 2009. 3

    2009

  26. [34]

    Fine-grained visual classi- fication of aircraft.arXiv preprint arXiv:1306.5151, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft.arXiv preprint arXiv:1306.5151, 2013. 2, 3, 6

    2013 arXiv

  27. [35]

    Alors: An algorithm recommender system.Artificial Intelligence, 244:291–314,

    Mustafa Mısır and Mich `ele Sebag. Alors: An algorithm recommender system.Artificial Intelligence, 244:291–314,

  28. [36]

    Discriminative models for information retrieval

    Ramesh Nallapati. Discriminative models for information retrieval. InProceedings of the 27th annual international ACM SIGIR conference on Research and development in in- formation retrieval, pages 64–71, 2004. 3

    2004

  29. [37]

    Leep: A new measure to evaluate transferabil- ity of learned representations

    Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferabil- ity of learned representations. InInternational Conference on Machine Learning, pages 7294–7305. PMLR, 2020. 2, 4, 6

    2020

  30. [38]

    Simple algorithm portfolio for sat.Artificial Intelligence Review, 40 (4):457–465, 2013

    Mladen Nikoli ´c, Filip Mari ´c, and Predrag Jani ˇci´c. Simple algorithm portfolio for sat.Artificial Intelligence Review, 40 (4):457–465, 2013. 6, 4

    2013

  31. [39]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 6, 3

    2008

  32. [40]

    A survey on transfer learn- ing.IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009

    Sinno Jialin Pan and Qiang Yang. A survey on transfer learn- ing.IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009. 1

    2009

  33. [41]

    Transferability estima- tion using bhattacharyya class separability

    Michal P ´andy, Andrea Agostinelli, Jasper Uijlings, Vitto- rio Ferrari, and Thomas Mensink. Transferability estima- tion using bhattacharyya class separability. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9172–9182, 2022. 2, 6, 4

    2022

  34. [42]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 6, 3

    2012

  35. [43]

    Metaood: Automatic selection of ood detection mod- els.arXiv preprint arXiv:2410.03074, 2024

    Yuehan Qin, Yichi Zhang, Yi Nian, Xueying Ding, and Yue Zhao. Metaood: Automatic selection of ood detection mod- els.arXiv preprint arXiv:2410.03074, 2024. 3, 4, 8

    2024 arXiv

  36. [44]

    Automatic classifier selection for non-experts.Pattern Analysis and Applications, 17(1): 83–96, 2014

    Matthias Reif, Faisal Shafait, Markus Goldstein, Thomas Breuel, and Andreas Dengel. Automatic classifier selection for non-experts.Pattern Analysis and Applications, 17(1): 83–96, 2014. 2

    2014

  37. [45]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019. 6

    1908 arXiv

  38. [46]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4510–4520, 2018. 6

    2018

  39. [47]

    Not all models are equal: Predicting model transferability in a self- challenging fisher space

    Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in a self- challenging fisher space. InEuropean Conference on Com- puter Vision, pages 286–302. Springer, 2022. 2, 4, 6

    2022

  40. [48]

    Analysis and pre- diction of nlp models via task embeddings.arXiv preprint arXiv:2112.05647, 2021

    Damien Sileo and Marie-Francine Moens. Analysis and pre- diction of nlp models via task embeddings.arXiv preprint arXiv:2112.05647, 2021. 2

    2021 arXiv

  41. [49]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015. 6

    2015

  42. [50]

    Rethinking the inception archi- tecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2818–2826, 2016. 6

    2016

  43. [51]

    Mnas- net: Platform-aware neural architecture search for mobile

    Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnas- net: Platform-aware neural architecture search for mobile. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 2820–2828, 2019. 6

    2019

  44. [52]

    Learning to learn: In- troduction and overview

    Sebastian Thrun and Lorien Pratt. Learning to learn: In- troduction and overview. InLearning to learn, pages 3–17. Springer, 1998. 2, 4

    1998

  45. [53]

    Transfer- ability and hardness of supervised classification tasks

    Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transfer- ability and hardness of supervised classification tasks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1395–1405, 2019. 2, 6, 4

    2019

  46. [54]

    Meta-learning: A survey.arXiv preprint arXiv:1810.03548, 2018

    Joaquin Vanschoren. Meta-learning: A survey.arXiv preprint arXiv:1810.03548, 2018. 2

    2018 arXiv

  47. [55]

    Ad- vances and challenges in meta-learning: A technical review

    Anna Vettoruzzo, Mohamed-Rafik Bouguelia, Joaquin Van- schoren, Thorsteinn R ¨ognvaldsson, and KC Santosh. Ad- vances and challenges in meta-learning: A technical review. IEEE transactions on pattern analysis and machine intelli- gence, 46(7):4763–4779, 2024. 2

    2024

  48. [56]

    A weighted correlation index for rankings with ties

    Sebastiano Vigna. A weighted correlation index for rankings with ties. InProceedings of the 24th international conference on World Wide Web, pages 1166–1176, 2015. 3, 6

    2015

  49. [57]

    How far pre-trained models are from neural collapse on the target dataset informs their trans- ferability

    Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre-trained models are from neural collapse on the target dataset informs their trans- ferability. InProceedings of the IEEE/CVF International 10 Conference on Computer Vision, pages 5549–5558, 20...

    2023

  50. [58]

    A survey of transfer learning.Journal of Big data, 3(1):9, 2016

    Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning.Journal of Big data, 3(1):9, 2016. 1

    2016

  51. [59]

    Wolpert and W.G

    D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization.IEEE Transactions on Evolutionary Com- putation, 1(1):67–82, 1997. 2

    1997

  52. [60]

    Listwise approach to learning to rank: theory and algorithm

    Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. InProceedings of the 25th international confer- ence on Machine learning, pages 1192–1199, 2008. 3

    2008

  53. [61]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 6, 8, 3

    2010

  54. [62]

    V-petl bench: A unified visual parameter- efficient transfer learning benchmark.Advances in neural information processing systems, 37:80522–80535, 2024

    Yi Xin, Siqi Luo, Xuyang Liu, Haodi Zhou, Xinyu Cheng, Christina E Lee, Junlong Du, Haozhe Wang, MingCai Chen, Ting Liu, et al. V-petl bench: A unified visual parameter- efficient transfer learning benchmark.Advances in neural information processing systems, 37:80522–80535, 2024. 1

    2024

  55. [63]

    A review on transferability estima- tion in deep transfer learning.IEEE Transactions on Artifi- cial Intelligence, 2024

    Yihao Xue, Rui Yang, Xiaohan Chen, Weibo Liu, Zidong Wang, and Xiaohui Liu. A review on transferability estima- tion in deep transfer learning.IEEE Transactions on Artifi- cial Intelligence, 2024. 1, 2

    2024

  56. [64]

    How transferable are features in deep neural networks?Ad- vances in neural information processing systems, 27, 2014

    Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks?Ad- vances in neural information processing systems, 27, 2014. 1

    2014

  57. [65]

    Logme: Practical assessment of pre-trained models for transfer learning

    Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. InInternational Conference on Ma- chine Learning, pages 12133–12143. PMLR, 2021. 1, 4, 6, 2

    2021

  58. [66]

    A survey on negative transfer.IEEE/CAA Journal of Automat- ica Sinica, 10(2):305–329, 2022

    Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer.IEEE/CAA Journal of Automat- ica Sinica, 10(2):305–329, 2022. 1

    2022

  59. [67]

    Model spider: Learning to rank pre- trained models efficiently.Advances in Neural Information Processing Systems, 36:13692–13719, 2023

    Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre- trained models efficiently.Advances in Neural Information Processing Systems, 36:13692–13719, 2023. 2, 3

    2023

  60. [68]

    A comprehensive survey on transfer learning.Proceedings of the IEEE, 109(1):43–76, 2020

    Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning.Proceedings of the IEEE, 109(1):43–76, 2020. 1 11 MetaRank: Task-Aware Metric Selection for Model Transferability Estimation Suppleme...

    2020

  61. [69]

    • We provide comprehensivetextual descriptionsof all datasets and MTE metrics in Section 7.2

    Supplementary In this supplementary, we include more details on the fol- lowing aspects: • We list the complete set of conventional meta-features ex- tracted using the PyMFE package [3] in Section 7.1. • We provide comprehensivetextual descriptionsof all datasets and MTE metri...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.