{"id":"e29d714c-bdeb-4ea7-99a4-02043d8ab05c","arxiv_id":"2412.19085","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DISCO scores pre-trained models for transfer by summing, over SVD spectral components, the product of each component's downstream-task performance and its singular-value proportion.","lead":"This paper introduces DISCO, a method that ranks pre-trained models for transfer learning by decomposing their extracted features into spectral components and weighting each component's task performance by its share of singular values. The method is fast, label-based, and reported to rank candidate models more accurately than existing selection metrics on classification and object detection benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Sg_ratio weighting in Eq. 6 rests on a spectral-concentration pattern observed on only three CNNs and three datasets; the paper's own ViT-extended benchmark shows average tau_omega dropping from 0.739 to 0.554 but never verifies the mechanism for transformers.","rationale":"The central empirical claim is that fine-tuning concentrates spectral energy on more transferable components; without that, Eq. 6 is an arbitrary weighting of per-component task scores. The paper's own Table 9 reveals a substantial drop when ViTs are added, which is consistent with the concern that the mechanism is CNN-specific. I also note a secondary reproducibility issue: each Z_g in Eq. 2 has rank s < d, so the class covariance Lambda_c in Eq. 7 is singular and Lambda_c^{-1} is undefined as written; a pseudo-inverse or regularization is needed. This does not change the verdict because it is likely fixable and is not the main threat to the SOTA claim on the reported benchmarks. The extended benchmark partially supports robustness, but it does not test the mechanism. Requiring the authors to release code and verify the spectral-concentration pattern on transformer backbones would settle the concern. Therefore the reader's CONDITIONAL verdict is appropriate.","tokens_in":17984,"tokens_out":9840,"duration_ms":98731,"concrete_test":"Fine-tune ViT-S, ViT-B, and Swin-T on Caltech101, Flowers, and Pets using the same protocol as Section 3.2, compute Sratio per spectral component (G=10) before and after fine-tuning, and overlay the curves on Figure 2. If the 'larger singular values increase in proportion' pattern is absent or reversed for these backbones, the Sg_ratio weighting in Eq. 6 is not a general transferability prior. Additionally, rerun Table 9's 24-model ranking using Scls = sum_g Sg_ncc without Sg_ratio; if the unweighted version matches the weighted version's tau_omega, the reported advantage is carried by Sg_task rather than by the spectral-distribution hypothesis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DISCO's only new ingredient relative to a per-component task score is the importance weight Sg_ratio, which is justified by the empirical claim that fine-tuning concentrates singular values onto larger spectral components (Fig. 2, Appendix B.1). That claim is established on exactly three CNN backbones (ResNet-152, DenseNet-201, Inception-v3) fine-tuned on exactly three datasets (Caltech101, Flowers, Pets). Eq. 6 then uses Sg_ratio computed on the original, pre-fine-tuning features as a universal transferability weight. If the concentration pattern is not a general property of fine-tuning—for example for ViT/Swin backbones, or for large domain gaps—the weighting in Eq. 6 is unjustified and DISCO could rank models incorrectly even when per-component task scores are accurate. Appendix B.5 does not close this gap: it only reports ranking correlations on a mixed 24-model pool containing three ViTs, and the average tau_omega drops from 0.739 (Table 1) to 0.554 (Table 9), indicating the method behaves differently once transformers are included. The spectral-concentration plot for those transformers is never shown, so the load-bearing premise of Eq. 6 remains unverified outside CNNs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DISCO, a transferability metric for pre-trained model selection. It performs SVD on features extracted by each candidate model, groups the singular values into spectral components, and scores each component with a task-specific measure (a nearest-centroid classifier score for classification, a linear-regression score for object detection) weighted by the component's singular-value share (Eq. 6, Eq. 10, Eq. 13). The authors motivate the weighting by an empirical observation that fine-tuning concentrates singular values onto larger spectral components (Fig. 2, App. B.1). DISCO is evaluated with weighted Kendall's tau against ground-truth fine-tuning results on 11 classification datasets for supervised and self-supervised CNN pools and on five object detection datasets with six YOLO variants, reporting average tau values of 0.739, 0.726, and 0.634, respectively, and is presented as achieving state-of-the-art model-selection performance.","tokens_in":18245,"tokens_out":6466,"duration_ms":61235,"significance":"If the empirical premise holds, DISCO is a simple, label-based, fine-tuning-free scoring method that extends naturally from classification to regression-style tasks, and the hard-example selection in Sec. 3.4 addresses practical scalability. The paper's strengths are its clean SVD decomposition, the task-specific score design, and the breadth of classification benchmarks, including both supervised and self-supervised model pools. The appendix's extended benchmark and tie-correction analysis are honest additions, but they also expose the main unresolved issue: the load-bearing spectral-concentration premise is verified only on a small set of CNNs and is not checked on transformers where the method's average correlation drops substantially.","major_comments":[{"comment":"The rating weight Sg_ratio in Eq. (6), which is the only new ingredient distinguishing DISCO from a plain sum of per-component task scores, is justified entirely by the empirical claim that fine-tuning concentrates singular values onto larger spectral components. That claim is demonstrated in Fig. 2 and App. B.1 on exactly three CNN backbones (ResNet-152, DenseNet-201, Inception-v3) and three datasets (Caltech101, Flowers, Pets). The extended benchmark in App. B.5 includes ViT-S, ViT-B, and Swin-T, but it reports only ranking correlations: the average tau drops from 0.739 in Table 1 to 0.554 in Table 9. Because the spectral-concentration diagnostic for transformers is never shown, the premise of Eq. (6) remains unverified outside the CNN family, and a failure of that premise could misrank models even when the per-component task scores are accurate. Please report the CF and Sratio plots for the three transformer models in App. B.5 and provide an ablation on the extended pool that replaces Sg_ratio with uniform weights or with architecture-specific weights.","section":"3.2, Eq. (6); Fig. 2; App. B.1 and B.5, Table 9"},{"comment":"The claimed state-of-the-art result is not supported by any uncertainty quantification or significance test. The average advantage over NCTI is only 0.013 on supervised models (0.739 vs. 0.726) and 0.007 on self-supervised models (0.726 vs. 0.719), and the per-dataset signs are mixed: for example, in Table 1 DISCO scores 0.575 on Pets where NCTI scores 0.924, and in Table 2 DISCO scores 0.542 on Pets where NCTI scores 0.805. Since each table entry appears to be a single deterministic computation, a paired bootstrap or permutation test across the 11 datasets would clarify whether the average gain is reliable. Without this, the headline 'state-of-the-art' claim is fragile.","section":"4.1, Tables 1 and 2"},{"comment":"The object-detection experiment is too narrow to support the paper's generality claim. The pool consists of six YOLOv5/YOLOv8 variants from the same detector family, all pre-trained on COCO, and the five datasets yield only 15 pairwise comparisons per dataset, making tau values and top-k probabilities coarse. The perfect Pr(top1)=1.000 on this small pool is not strong evidence that the regression score in Eq. (12) generalizes to diverse detection architectures. Please extend the pool to at least one non-YOLO family (e.g., Faster R-CNN or a transformer detector) or temper the claim of generality accordingly.","section":"4.3, Table 3"},{"comment":"The complexity statement for naive SVD is incorrect. For a matrix Z in R^{N x d}, standard SVD complexity is O(N d^2) when N > d, not O(N^3 d) as stated. After PCA reduction to d=128 in App. A.2, the relevant tall-skinny regime indeed gives O(N d^2). The empirical runtime results in Fig. 4d and Table 5 may still be valid, but the analytical motivation for hard-example selection should be corrected and re-derived.","section":"3.4"}],"minor_comments":[{"comment":"The main experiments use G=10 (Sec. 3.2), but Sec. 4.4 states that G=6 or G=8 'offer the best balance.' Please clarify whether Tables 1-3 use G=10 throughout and whether the reported averages change if the recommended G is used; this affects how the reader interprets the group-number sensitivity.","section":"3.2 and 4.4, Fig. 4c"},{"comment":"The tie-correction analysis considers models within 0.1% accuracy of each other as tied, but it reports only DISCO vs. SFDA and LogME, omitting NCTI and ETran, which are the relevant competitors in Tables 1 and 3. Please include those baselines or explain the omission.","section":"App. B.6, Table 10"},{"comment":"Several small typographical errors should be fixed, including 'framawork' in the caption of Fig. 4a and 'featrue' in the text surrounding Eq. (16).","section":"Throughout"},{"comment":"The object-detection ground-truth table does not state the training protocol (epochs, batch size, augmentation, hyperparameter sweep) used to obtain the mAP50 values. Adding these details would make the benchmark reproducible.","section":"4.3 and App. B.4, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is interesting, but the load-bearing spectral-concentration premise needs direct evidence on transformer architectures, and the headline SOTA claim needs statistical support. If the requested diagnostics and significance tests are provided, the paper could be publishable; in its current form, the evidence is not yet conclusive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes DISCO, a model selection metric that decomposes features into spectral components via SVD, scores each component with a task-specific estimator (nearest-centroid for classification, linear regression for detection), and weights the component scores by singular value proportions. That decomposition-based weighting is a real addition to a literature that mostly analyzes whole features. The method is clearly specified enough to reimplement, and the experimental scope is broader than most: 11 classification datasets, supervised and self-supervised CNN pools, an object-detection benchmark, ablations on group number, a hard-example sampling scheme, and an extended benchmark in the appendix.\n\nWhat it does well: the gains on the main supervised benchmark are consistent (average tau 0.739 vs NCTI's 0.726), with larger margins on fine-grained datasets like Aircraft and Cars. The object-detection result (0.634 vs Etran's 0.522) is strong, though the pool is small—6 YOLO variants on 5 datasets. The appendix also contains a thoughtful re-analysis that treats accuracy differences within 0.1% as ties, and DISCO still wins.\n\nThe soft spots are real but not fatal. The weighting in Eq. 6 rests on the empirical claim that fine-tuning concentrates singular values onto larger spectral components, but that observation is shown for exactly three CNN backbones and three datasets. The extended benchmark in Appendix B.5 includes ViT/Swin models, and DISCO still ranks the mixed 24-model pool best, but the average tau drops from 0.739 to 0.554 and the spectral-concentration plots for transformers are never shown. So the mechanism behind the weighting is unverified outside CNNs, even if the method still works empirically. Relatedly, the main comparison has no error bars or significance tests; a 1.8% relative improvement over NCTI could be noise. There is also an internal inconsistency: Section 3.2 says G=10 is used, while Section 4.4 says G=6 or 8 gives the best balance, leaving unclear which setting produced the headline numbers. Minor technical issue: the claimed O(N^3 d) complexity for SVD is wrong; for an N x d matrix it is O(N d^2) when N > d.\n\nOn the reader's circularity concern: I don't think it lands. The Sg_ratio weights are computed from the original features, not from fine-tuning outcomes, so the metric is not circular in a damaging way.\n\nWho this is for: anyone working on pre-trained model selection, transferability estimation, or model-hub tooling. It deserves serious refereeing. If I were the editor I would ask for error bars or significance tests, a clear statement of which G is used and why, and either a spectral analysis for ViTs or a softened claim about the generality of the concentration pattern.","headline":"DISCO adds a genuinely new SVD-based transferability metric with solid empirical wins, but the core weighting premise is verified only on CNNs and the SOTA claims rest on thin margins.","tokens_in":18767,"tokens_out":2471,"would_cite":true,"duration_ms":24232,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained models can be ranked for transfer learning by the distribution of their features' singular values, weighted by each spectral component's fit to target labels.","keywords":["transfer learning","pre-trained model selection","transferability estimation","singular value decomposition","spectral components","model hub","fine-tuning","ranking correlation"],"falsifier":"Compute $S_g^{\\mathrm{ratio}}$ before and after fine-tuning for a model pool that includes vision transformers or large domain shifts; if the singular-value proportion does not consistently increase for the largest components, or if the DISCO ranking's weighted Kendall's tau against true fine-tuning performance drops to chance levels, the central claim fails.","tokens_in":17788,"feed_emoji":"🧠","tokens_out":4570,"duration_ms":39852,"temperature":0.7,"pith_summary":"Pre-trained models in a model hub differ wildly in how well they transfer to a new task, and fine-tuning every candidate to find out is expensive. This paper claims that the fine-tuning process itself leaves a fingerprint in the singular-value spectrum of the extracted features: after fine-tuning, the singular values concentrate on a few spectral components, and those components are the transferable ones. The authors build a score, DISCO, that weights each spectral component by its ratio of singular values and its estimated performance on the target labels, and they report that this score ranks pre-trained models better than existing transferability metrics on supervised classification (average weighted Kendall's tau 0.739), self-supervised classification (0.726), and object detection (0.634). If the claim holds, model selection becomes a fast forward pass plus a cheap linear or centroid estimate instead of repeated fine-tuning.","feed_headline":"Singular values pick the best pre-trained model without fine-tuning","feed_subtitle":"DISCO weights spectral components by singular-value share and label fit, topping prior metrics on three benchmarks.","key_machinery":"The central object is the SVD of the feature matrix $Z = U\\Sigma V^T$ of target data extracted by each pre-trained model, with singular values grouped into $G$ spectral components. The load-bearing identity is the score $S_{\\mathrm{DISCO}} = \\sum_g S_g^{\\mathrm{task}} S_g^{\\mathrm{ratio}}$, where $S_g^{\\mathrm{ratio}}$ is the proportion of total singular values in component $g$ and $S_g^{\\mathrm{task}}$ is a task-specific transferability estimate (nearest-centroid confidence for classification, negative MSE of a linear-regression pseudo-inverse for regression). Hard-example selection via LDA reduces the sample size so the SVD stays cheap. This machinery does the work of turning the empirical concentration observation into a ranking criterion.","core_discovery":"The central claim is that a pre-trained model's transferability to a downstream task is determined by the distribution of spectral components of its extracted features, not by properties of the whole feature set. The paper observes, through SVD of features before and after fine-tuning, that fine-tuning concentrates the singular-value distribution onto components with larger singular values, and that these components transfer better. DISCO formalizes this as $S_{\\mathrm{DISCO}} = \\sum_{g=1}^{G} S_g^{\\mathrm{task}} \\cdot S_g^{\\mathrm{ratio}}$, with $S_g^{\\mathrm{task}}$ measuring how well the $g$-th component separates target classes (nearest-centroid confidence) or fits target regression labels (linear least squares), and $S_g^{\\mathrm{ratio}}$ the share of total singular values in that component. The paper reports that this score achieves the highest average weighted Kendall's tau among compared metrics on all three benchmarks.","pith_inferences":["If the spectral-concentration pattern holds for transformer backbones, DISCO could be applied to large model hubs of foundation models essentially unchanged, since it only needs features and labels.","The per-component weighting suggests a testable extension: instead of choosing one model, one could blend spectral components across models, selecting the best component per group to form an ensemble.","The hard-example sampling insight, that hard examples carry more information about transferability, could be reused by other transferability metrics, since the paper shows random sampling loses more accuracy at equal sample sizes."],"forward_implications":["Model hubs can be ranked with a single forward pass plus a lightweight classifier, avoiding per-model fine-tuning.","The same framework works for classification and regression, so object detection and other multi-task settings can share one selection criterion.","Self-supervised models, which lack pre-trained classifiers, can still be ranked because the score uses only features and target labels.","With hard-example selection, rankings remain strong at 20 to 40 percent of the data, making the method practical for large target datasets.","The score is a weighted sum of interpretable per-component terms, so a model's transferability profile can be inspected rather than reduced to a single black-box number."],"supporting_citations":[{"why":"Supplies the fine-tuning settings and ground-truth accuracy tables for the supervised and self-supervised model pools, and the SFDA baseline that DISCO must beat.","marker":"Shao et al. 2022"},{"why":"The NCTI baseline that DISCO compares against; its neural-collapse distance idea is the prior state of the art that DISCO supersedes.","marker":"Wang et al. 2023"},{"why":"LogME, the maximum-likelihood baseline used for both supervised and self-supervised comparisons.","marker":"You et al. 2021"},{"why":"ETran, the energy-based baseline that supplies the regression-score variant (Slmr) used in the object detection comparison.","marker":"Gholami et al. 2023"},{"why":"LEEP, a label-fitness baseline that DISCO extends; the NLEEP variant is also used in comparisons.","marker":"Nguyen et al. 2020"}],"fun_headline_variants":["Spectral component shares reveal best pre-trained model without fine-tuning","DISCO ranks pre-trained models by label-aware singular-value distribution","Transferability lives in spectral component distribution, not whole features","SVD spectrum weights decide which pre-trained model transfers best"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the observed tendency of fine-tuning to concentrate singular values onto the largest spectral components, measured on three CNN architectures and three image datasets, is a general property of fine-tuning across architectures, tasks, and domain gaps.","fun_headline_variants_meta":{"raw":{"variants":["Spectral component shares reveal best pre-trained model without fine-tuning","DISCO ranks pre-trained models by label-aware singular-value distribution","Transferability lives in spectral component distribution, not whole features","SVD spectrum weights decide which pre-trained model transfers best"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1580,"prompt_tokens":950,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":561}},"tokens_in":566,"tokens_out":630,"duration_ms":7256,"temperature":1.0,"reasoning_tokens":561,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:19.684858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $S_g^{\\mathrm{ratio}}$ before and after fine-tuning for a model pool that includes vision transformers or large domain shifts; if the singular-value proportion does not consistently increase for the largest components, or if the DISCO ranking's weighted Kendall's tau against true fine-tuning performance drops to chance levels, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fine-tuning settings and ground-truth accuracy tables for the supervised and self-supervised model pools, and the SFDA baseline that DISCO must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The NCTI baseline that DISCO compares against; its neural-collapse distance idea is the prior state of the art that DISCO supersedes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LogME, the maximum-likelihood baseline used for both supervised and self-supervised comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ETran, the energy-based baseline that supplies the regression-score variant (Slmr) used in the object detection comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LEEP, a label-fitness baseline that DISCO extends; the NLEEP variant is also used in comparisons."}],"review_version":1}