{"id":"4aa2c09f-4665-49e7-a7f4-3b5faaceecb1","arxiv_id":"2506.14473","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RAM-APL combines distance rankings and pseudo-class label accuracy from two foundation models to select training subsets, outperforming twelve baselines on fine-grained image datasets.","lead":"Researchers tested whether off-the-shelf foundation models can pick the most useful training images for fine-grained classification, and found they often beat hand-trained extractors on fine-grained datasets. They then built RAM-APL, which combines CLIP and DINOv2 rankings and pseudo-class scores, and report higher accuracy than standard subset selection baselines on pet, food, and bird datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA and FM-dominance claims rest on 10-epoch-trained traditional IEs; a converged extractor could close the gap.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: traditional IE baselines are trained for only 10 epochs, which may not represent the traditional extractors the paper claims to outperform. I agree and judge it the single most load-bearing issue because it affects Observation 2 and all baseline comparisons across Pet, Food-101, and CUB, rather than only one dataset. The Pet hyperparameter-selection issue is real but secondary: alpha and beta were tuned on Pet, yet the same fixed values are used on Food-101 and CUB, so those results provide partial independent support for the method. I also credit the released code, the clear method description, and the cross-architecture experiment (Table 6) as genuine supporting evidence. However, the convergence question is unresolved and directly determines whether the headline claims are overstated. Because the reader's CONDITIONAL verdict already hinges on this concern, my read does not change the verdict; it reinforces the condition. The recommended check—retraining all traditional IE baselines to convergence—is a single, well-defined experiment that would settle the concern.","tokens_in":23365,"tokens_out":4505,"duration_ms":49439,"concrete_test":"Retrain every baseline extractor in Tables 8–10 (and the model-TD arm of the single-model study in Figure 6) with the same backbone and optimizer but 90 epochs, or until convergence, on the target training set, then rerun subset selection at all reported sampling rates. If RAM-APL no longer beats every baseline at every rate, or if a fully trained model-TD outperforms all foundation models in the single-model study, the SOTA claim and Observation 2 both need to be weakened. If RAM-APL still wins with the same margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claims—Observation 2 ('FMs consistently outperform traditional IEs') and the SOTA comparison in Tables 8–10—are both evaluated against traditional information extractors trained for only 10 epochs on the target dataset. Section 6.1 states this explicitly: baselines (2)–(12) use a model trained on the target training set for 10 epochs. Appendix A.1 likewise defines 'model-TD' as a 10-epoch pretrained model. This is far below the training regimes of representative traditional extractors in the literature: the paper itself notes in Section 2 that TDDS uses 90 epochs on ImageNet-1K to obtain training dynamics. The 10-epoch choice may reflect a compute-saving strategy (Guo et al., 2022), but it is not established that such a lightly trained extractor is a faithful representative of 'traditional IEs.' If a converged traditional extractor produces significantly better feature/gradient/dynamics signals, the observed FM advantage could shrink or reverse on fine-grained datasets, and the claim that RAM-APL beats every baseline at every sampling rate would no longer be supported. This is not an internal inconsistency, but a correctness risk in the comparison protocol: the independent variable (IE type) is confounded with IE training budget, and the burden of showing that 10 epochs is sufficient for the baselines' extractors is not met.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies one-shot subset selection for image classification, asking whether foundation models (FMs) can replace traditional information extractors (IEs) and whether all FMs perform equally. It reports that FMs consistently outperform traditional IEs on fine-grained datasets, while their advantage is limited on coarse-grained noisy datasets. Motivated by these observations, the authors propose RAM-APL, which fuses multiple FMs by combining a within-class distance ranking mean (RAM) and a pseudo-class label accuracy score (APL) into a weighted selection score. The method is evaluated on Oxford-IIIT Pet, Food-101, and CUB-200-2011, where it is reported to outperform 12 baselines at all tested sampling rates, with average gains over Random of 3.74%, 4.44%, and 6.40% respectively.","tokens_in":23611,"tokens_out":4729,"duration_ms":43506,"significance":"If the results hold, the paper offers a practical, dataset-agnostic alternative to traditional IEs for fine-grained subset selection and provides a useful empirical mapping of when FMs help. The paper ships its code, uses multiple datasets and seeds, and includes cross-architecture and ablation experiments. The main claims, however, rest on a comparison protocol whose fairness and statistical reliability need to be established before the SOTA and FM-dominance conclusions can be accepted.","major_comments":[{"comment":"The comparison with traditional information extractors uses IEs trained for only 10 epochs on the target dataset. This is explicitly stated for baselines (2)-(12) in Section 6.1 and for model-TD in Appendix A.1. For training-dynamics methods such as Forgetting and GraNd, 10 epochs is too short to obtain reliable dynamics, and the paper itself notes in Section 2 that TDDS uses 90 epochs on ImageNet-1K. The independent variable (IE type) is therefore confounded with training budget, and the claim that 'FMs consistently outperform traditional IEs' (Observation 2) and the SOTA claims in Tables 8-10 are not yet supported against properly converged traditional IEs. I request additional experiments with traditional IEs trained to convergence (or at least to a comparable budget) to confirm that the FM advantage persists.","section":"Section 6.1 and Appendix A.1"},{"comment":"The hyperparameters α=0.2, β=1 and the FM pair {CLIP, DINOv2} are selected using the Pet dataset (Figure 4, Table 2) and then applied to Food-101 and CUB. This makes the reported gains on the other two datasets a form of tuning-on-one-dataset evaluation rather than independent predictions. I acknowledge that the hyperparameters are few and the method is conceptually generic, but the selection of the FM pair is a dataset-dependent design choice that should be treated as a validation process. The authors should either report results with per-dataset validation, show robustness to a small grid of hyperparameters on all datasets, or clearly state that the reported cross-dataset numbers are the result of transferring configuration chosen on Pet and therefore carry a selection bias.","section":"Section 6.2 and Section 6.4"},{"comment":"The claim that RAM-APL 'outperforms all baselines at each sampling rate' is not supported by the reported variance at several points. For example, in Table 10 at the 30% sampling rate, RAM-APL reports 42.6±3.1, while GC reports 36.1±10.4 and Cal reports 32.7±4.5; these confidence intervals overlap substantially. The averaged improvement over Random of 6.40% on CUB is heavily influenced by this high-variance point. The paper should report significance tests or at least a discussion of which differences are reliable, given the small number of seeds (three for Food-101 and CUB). Without this, the 'superior' in the title and contributions is stronger than the evidence supports.","section":"Tables 8-10, Section 6.2"}],"minor_comments":[{"comment":"The main text states that the single-model study is conducted on five datasets, while Appendix A describes results on nine datasets; please reconcile this discrepancy.","section":"Section 4 (Experimental Setting)"},{"comment":"The symbol S is used both for the subset selected by the algorithm and for the index set of a class, which is confusing; please rename one of them.","section":"Section 5.2, Equation (1)"},{"comment":"Equation (3) defines Rc for a class, but Equation (7) uses R without a class index; please clarify that R denotes the vector obtained by concatenating ranking means over all classes.","section":"Section 5.2, Equations (3) and (7)"},{"comment":"There is a missing space in 'β = 1for our method'; please correct the typo.","section":"Section 6.2"},{"comment":"The caption of Table 2 defines D, C, S, E but the table uses bullet symbols; please make the legend self-contained or move it into the caption text.","section":"Section 6.4 and Table 2"},{"comment":"The model name 'SigLIP-base-patch16-22' appears to be a typo for 'patch16-224'; please verify.","section":"Appendix A.1"},{"comment":"The claim of being 'the first to investigate selection with multiple foundation models' is strong; please soften it or cite related ensemble-based selection work to avoid overclaiming.","section":"Section 5 (overall)"},{"comment":"The 'frequency of being the best IE' metric is a coarse categorical count; it would be informative to also report the average accuracy gap between the best FM and the best traditional IE to quantify the magnitude of the claimed advantage.","section":"Section 4, Figure 1(c)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely topic and contains a substantial set of experiments, but the headline claims rest on a comparison protocol that is not yet convincing. The 10-epoch training of traditional IEs is the biggest risk; adding converged baselines and a small statistical analysis would substantially strengthen the manuscript. The tuning-on-Pet issue is also important, though standard in this literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful paper. The single-model study across nine datasets is the first thorough look I know at when foundation models help as information extractors for one-shot subset selection, and the observations—FMs beat traditional IEs on fine-grained, advantage fades on coarse noisy, and FM quality on the downstream task does not predict IE quality—are worth having. The RAM-APL score (ranking mean across models plus pseudo-class accuracy) is simple and clearly motivated, and the gains on Food-101 and CUB with fixed hyperparameters look real. I'd cite it.\n\nThe soft spots are two. First, the FM-dominance claim (Observation 2) is tested only against traditional IEs trained for 10 epochs on the target dataset. That is a short budget for a 75k-image dataset like Food-101. The paper itself notes that TDDS uses 90 epochs on ImageNet-1K. If a converged extractor closes the gap, the SOTA story weakens. This is not an internal contradiction, but it is a confounding variable in the comparison protocol. The fix is simple: train at least one traditional IE to convergence and rerun the key tables. Second, the Pet headline result uses alpha, beta, and the FM pair selected on Pet itself. The Food-101 and CUB results are clean, but the Pet numbers are partially in-sample. That should be acknowledged or re-validated with a small holdout.\n\nOn the math: the RAM and APL equations are straightforward and match the text. The paper is honest about its settings. I don't see a circularity problem—the selection score uses features and labels, not fitted benchmark results.\n\nWho is this for? Anyone working on data-efficient training or coreset selection for fine-grained classification. It deserves a serious referee; the empirical study alone justifies that. My recommendation: engage it, but ask for the converged-traditional-IE experiment and for a Pet result with hyperparameters chosen without Pet. Both are cheap relative to the claims they support.","headline":"A practical FM-based subset selection method with honest observations, but the FM-vs-traditional comparison and the Pet headline numbers both rest on choices that need one more experiment.","tokens_in":24147,"tokens_out":2807,"would_cite":true,"duration_ms":32348,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Foundation models can replace dataset-trained extractors in one-shot subset selection, and a CLIP+DINOv2 score beats all twelve baselines on fine-grained benchmarks.","keywords":["one-shot subset selection","coreset selection","foundation models","fine-grained image classification","RAM-APL","CLIP","DINOv2","data-efficient training"],"falsifier":"Rerun the Single-Model Study with a traditional extractor trained to convergence (for example, the same ResNet backbone trained on the target set for 90 epochs instead of 10) and compare MIN, KCG, GC, and MDS selections at 10%, 30%, and 50% sampling rates on Pet and CUB; if the fully trained extractor matches or beats CLIP and DINOv2 in the accuracy of models trained on the selected subsets, Observation 2 is falsified.","tokens_in":23130,"feed_emoji":"🎯","tokens_out":8280,"duration_ms":74429,"temperature":0.7,"pith_summary":"This paper asks whether foundation models can replace traditional information extractors in one-shot subset selection, where a small informative subset is chosen in a single pass and then used to train a target model. Through a single-model study across nine datasets, it reports that foundation models consistently outperform extractors pre-trained on the target dataset for ten epochs on fine-grained datasets, clean or noisy, while the advantage largely disappears on coarse-grained datasets with noisy labels. The paper then proposes RAM-APL, which combines features from CLIP and DINOv2 without aligning their feature spaces, scoring each sample by a weighted mean of its within-class distance ranking and its pseudo-class label inaccuracy, $\\mathrm{Score} = W_1 R + W_2(1-\\varphi)$. With weights $W_1$ and $W_2$ set by $\\alpha=0.2$ and $\\beta=1$, the method beats Random by 3.74 points on Pet, 4.44 points on Food-101, and 6.40 points on CUB on average across sampling rates, and outperforms all twelve baselines at every tested rate. If the claims hold, subset selection no longer needs a model trained on the full target dataset, which removes a major source of dataset dependency and pre-training cost.","feed_headline":"Two frozen models beat 12 subset-selection baselines","feed_subtitle":"RAM-APL combines CLIP and DINOv2 rankings to lift fine-grained accuracy by up to 6.4 percent over random selection.","key_machinery":"The mechanism is the score rule $\\mathrm{Score} = W_1 R + W_2(1-\\varphi)$, built from two per-sample metrics computed separately in each foundation model's feature space. RAM ('ranking mean') computes the Euclidean distance of each sample to its class-mean feature, ranks samples within the class, and averages the normalized ranks across all foundation models, so a low value means the sample is close to the class prototype in every model's view. APL ('accuracy of pseudo-class labels') assigns each sample the class label of its nearest class-mean feature in each model's space, then averages the indicator that this pseudo-label equals the true label; $1-\\varphi$ is high for samples that different models keep confusing with other classes. The weights follow a logistic schedule in the sampling rate $p$: $W_1 = \\alpha + (1-\\alpha)/(1+e^{\\beta(p-0.5)})$ and $W_2 = 1-W_1$, with $\\alpha=0.2$ and $\\beta=1$, so $W_1$ stays above $W_2$ for all tested sampling rates while the inter-class term gains influence as $p$ grows. The method selects the smallest-score samples up to the budget.","core_discovery":"The central claim is that a foundation model used as a frozen feature extractor is a better information extractor for one-shot subset selection on fine-grained image datasets than a small model pre-trained on the target training set for ten epochs, and that this advantage is task-dependent: it holds on fine-grained datasets with clean or noisy labels but mostly vanishes on coarse-grained datasets with noisy labels. A second claim is that no single foundation model is always the best extractor; the best FM varies with the dataset, the sampling rate, and the selection algorithm, so the paper's method avoids choosing one. RAM-APL therefore uses two foundation models, CLIP and DINOv2, and converts their unaligned features into two comparable signals per sample: the average rank of the sample's distance to its class-mean feature across models, and the fraction of models whose nearest-class-center pseudo-label matches the ground truth. The final score is $W_1 R + W_2(1-\\varphi)$ with weights that keep the intra-class ranking dominant at all sampling rates while letting the inter-class term grow as the budget increases. On Pet, Food-101, and CUB the paper reports that this score selects subsets whose trained models reach state-of-the-art accuracy at every tested sampling rate.","pith_inferences":["Because RAM and APL need only class-mean features and nearest-center pseudo-labels, the same score could in principle be computed without any ground-truth labels by using pseudo-labels as the target; the paper does not test this, but it suggests a path to subset selection in unlabeled or weakly labeled pools.","The two signals decompose data difficulty in a testable way: RAM measures within-class representativeness and APL measures cross-class confusability, so one could predict that RAM-APL's advantage grows as class boundaries become visually subtler and shrinks as classes become easy to separate.","Given the near-zero cosine similarity between features of different foundation models reported in the paper, adding a third model should help only if it supplies a genuinely new visual bias; otherwise multi-model gains should saturate.","The paper's noisy fine-grained results, including Pet with 20% and 40% label noise, suggest ranking-based selection is robust to label noise when classes are fine-grained, which could be tested directly on other noisy fine-grained benchmarks."],"forward_implications":["If the finding holds, one-shot subset selection can drop the target-dataset pre-training step entirely, since frozen CLIP and DINOv2 features replace the traditional extractor.","Because RAM-APL beats every single foundation model in its own experiments, practitioners no longer need to search for the best extractor; the multi-model consensus score is the default.","The reported average gains over Random, 3.74 points on Pet, 4.44 on Food-101, and 6.40 on CUB, imply the largest benefits on fine-grained recognition tasks with many visually similar classes.","Cross-architecture results on Pet show subsets chosen with CLIP+DINOv2 improve a MobileNet-V3 target model as well as a ResNet target, so the selected data are not tied to one architecture.","The method is not a universal fix: on coarse-grained datasets with noisy labels the FM advantage diminishes, so the engineering recommendation is to apply it where inter-class differences are subtle."],"supporting_citations":[{"why":"CLIP is one of the two frozen foundation-model feature extractors used by RAM-APL.","marker":"(Radford et al., 2021)"},{"why":"DINOv2 is the other frozen feature extractor in the default RAM-APL pair.","marker":"(Oquab et al., 2023)"},{"why":"Earlier study finding that simply using a foundation model for subset selection does not guarantee better performance, which this paper directly investigates.","marker":"(Xie et al., 2023)"},{"why":"Provides the ten-epoch pre-training protocol used to make the traditional information-extractor baselines.","marker":"(Guo et al., 2022)"},{"why":"K-Center Greedy is one of the classical feature-based subset selection baselines the FM pipeline is compared against.","marker":"(Sener & Savarese, 2017)"},{"why":"Moderate DS provides the distance-to-class-center selection baseline that RAM's ranking metric extends and compares against.","marker":"(Xia et al., 2023)"},{"why":"Oxford-IIIT Pet is one of the three fine-grained benchmarks where state-of-the-art results are claimed and the dataset for ablations.","marker":"(Parkhi et al., 2012)"},{"why":"Food-101 is the second fine-grained benchmark used for the main comparison.","marker":"(Bossard et al., 2014)"},{"why":"Caltech-UCSD Birds-200-2011 is the third fine-grained benchmark where the largest reported gain over Random is 6.40 points.","marker":"(Wah et al., 2011)"},{"why":"Supplies the idea that easier samples facilitate optimization first, motivating the sampling-rate-dependent weight schedule.","marker":"(Swayamdipta et al., 2020)"}],"fun_headline_variants":["Frozen CLIP and DINOv2 excel at one-shot subset selection","Two foundation models choose better training subsets","Multi-FM scoring lifts fine-grained subset selection","RAM-APL: combine CLIP and DINOv2 for smarter picks","Foundation models beat task-specific extractors on fine-grained data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model pre-trained on the target training set for just ten epochs adequately represents what traditional information extractors achieve, because several published extractors are trained far longer (the paper itself notes one baseline needing 90 epochs on ImageNet-1K); if a fully converged traditional extractor closed the gap with foundation models, the claimed FM advantage on fine-grained datasets would weaken.","fun_headline_variants_meta":{"raw":{"variants":["Frozen CLIP and DINOv2 excel at one-shot subset selection","Two foundation models choose better training subsets","Multi-FM scoring lifts fine-grained subset selection","RAM-APL: combine CLIP and DINOv2 for smarter picks","Foundation models beat task-specific extractors on fine-grained data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2135,"prompt_tokens":1010,"completion_tokens":1125,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":1041}},"tokens_in":626,"tokens_out":1125,"duration_ms":10414,"temperature":1.0,"reasoning_tokens":1041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:47.511939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Single-Model Study with a traditional extractor trained to convergence (for example, the same ResNet backbone trained on the target set for 90 epochs instead of 10) and compare MIN, KCG, GC, and MDS selections at 10%, 30%, and 50% sampling rates on Pet and CUB; if the fully trained extractor matches or beats CLIP and DINOv2 in the accuracy of models trained on the selected subsets, Observation 2 is falsified.","supporting_citations":[{"cited_title":"Towards free data selection with general-purpose models","cited_arxiv_id":null,"evidence_quote":"Earlier study finding that simply using a foundation model for subset selection does not guarantee better performance, which this paper directly investigates."},{"cited_title":"DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning","cited_arxiv_id":"2204.08499","evidence_quote":"Provides the ten-epoch pre-training protocol used to make the traditional information-extractor baselines."},{"cited_title":"Moderate coreset: A universal method of data selection for real-world data-efficient deep learning","cited_arxiv_id":null,"evidence_quote":"Moderate DS provides the distance-to-class-center selection baseline that RAM's ranking metric extends and compares against."},{"cited_title":"M., Vedaldi, A., Zisserman, A., and Jawahar, C","cited_arxiv_id":null,"evidence_quote":"Oxford-IIIT Pet is one of the three fine-grained benchmarks where state-of-the-art results are claimed and the dataset for ablations."},{"cited_title":"Food-101--mining discriminative components with random forests","cited_arxiv_id":null,"evidence_quote":"Food-101 is the second fine-grained benchmark used for the main comparison."},{"cited_title":"The caltech-ucsd birds-200-2011 dataset","cited_arxiv_id":null,"evidence_quote":"Caltech-UCSD Birds-200-2011 is the third fine-grained benchmark where the largest reported gain over Random is 6.40 points."}],"review_version":1}