{"id":"30db7be7-5656-4ab8-bd7e-0b5f38efb0fd","arxiv_id":"2504.16277","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DataS^3 is a new benchmark demonstrating that deployment-specific data subsets can outperform full-pool training, with up to 51.3% accuracy gains, while current methods fall short.","lead":"This paper introduces DataS^3, a benchmark for selecting training data subsets that specialize models for specific deployments, using five real-world datasets and 12 deployment scenarios. It shows that oracle subsets curated with domain knowledge can beat training on all data by up to 51.3%, while existing general-purpose subset selection baselines often fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AutoArborist ID/OOD definitions directly contradict across §4.3 and §D.3, so the flagship 51.3% expert-subset gain is not reproducible as stated.","rationale":"The reader's weakest assumption centers on the oracle-like nature of the expert subsets, which use privileged information such as GPS coordinates and deployment class lists. That concern is real, but the paper discloses this privileged information in §4, so it is an acknowledged upper-bound construction rather than a hidden flaw. The more load-bearing issue is the AutoArborist ID/OOD contradiction: the two sections of the manuscript cannot both be correct, and the choice between them changes what the expert subset is and what the 51.3% gain means. Because the central claim is an existence claim supported by the largest gain in Table 1, an unreproducible protocol for that gain is the single weakest point. The released code and splits make this checkable, so the appropriate response is to keep the conditional verdict and require the authors to resolve the contradiction and re-verify the affected rows. I do not see a basis for rejection, because the benchmark resource and other datasets may still stand after correction.","tokens_in":25515,"tokens_out":7327,"duration_ms":76554,"concrete_test":"Download the released AutoArborist splits from datas3-benchmark.github.io and count training-pool images per deployment city. This single count determines whether Surrey/Calgary or LA/DC are truly in-distribution, resolving the §4.3 versus §D.3 contradiction. Then recompute the four AutoArborist expert-subset accuracies under the corrected ID/OOD assignment; if any value in the AutoArborist rows of Table 1 shifts materially, the headline 51.3% gain must be re-stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central existence claim—that deployment-specific subsets can outperform training on all data—is supported most dramatically by AutoArborist Deployment 1 (Surrey), where the knowledge-driven subset reaches 0.86 accuracy versus 0.35 for all-data, a relative improvement of 51.3%. That result depends on what the expert subset actually contains, but the paper gives two mutually exclusive descriptions. Section 4.3 states that Surrey and Calgary are in-distribution (ID) deployments with some of these cities' data in the training pool, and that Washington DC and Los Angeles are out-of-distribution (OOD) with no city data in the pool. Appendix D.3 states the opposite: LA and DC are ID with city data in the training pool, while Surrey and Calgary are OOD with no city data. The knowledge-driven subset description in §4.3—using Surrey/Calgary data for those deployments and San Francisco/San Jose for LA—is compatible with the first reading but not the second. If Surrey is ID, its expert subset is the city's own training images, making the comparison largely one of in-distribution city data versus a huge pool of other cities; if Surrey is OOD, the subset is a nearest-city proxy. These are materially different selection rules, and Table 1 gives no way to tell which one produced the reported gains. This is not a disclosure about oracle information; it is an internal inconsistency in the experimental protocol, and it directly affects the largest headline number in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes dataset subset selection for specialization (DS3): given a general training pool and a small query set from a deployment-specific distribution, select a training subset that maximizes deployment performance. It introduces DataS3, a benchmark spanning five datasets (iWildCam, AutoArborist, GeoDE, NuScenes, FishDetection) with twelve deployments, evaluates a suite of coreset and data-filtering baselines, and reports proof-of-concept expert/knowledge-driven subsets that often outperform training on all available data. The central empirical claim is that deployment-specific curated subsets can substantially beat all-data training while using far less data.","tokens_in":25851,"tokens_out":9373,"duration_ms":81505,"significance":"If the protocol issues are resolved, DataS3 would be a timely and useful benchmark: it operationalizes a practical problem that existing subset-selection benchmarks do not target, and the existence result that curated subsets can outperform training on the full pool at a fraction of the data size is nontrivial and motivating for new methods. The paper also ships open-source splits and code, which strengthens its reproducibility. However, the current draft contains direct internal contradictions in the experimental protocol, and several headline numbers in the results section are not supported by the reported tables. The benchmark's value can only be assessed after these load-bearing issues are corrected.","major_comments":[{"comment":"The paper gives two mutually exclusive descriptions of the AutoArborist deployment splits. Section 4.3 states that Surrey and Calgary are in-distribution (ID) deployments with some of their data in the training pool, while Los Angeles and Washington DC are out-of-distribution (OOD) with no city data in the pool; the knowledge-driven subsets are then built from Surrey/Calgary data for Deployments 1 and 2 and from nearby cities for Deployments 3 and 4. Appendix D.3 states the opposite, identifying Los Angeles and Washington DC as ID and Surrey and Calgary as OOD, and describes the expert subsets accordingly. These two protocols produce materially different subsets, and the largest headline result (AutoArborist Deployment 1, 0.86 vs 0.35) depends on which description is correct. Please unify the two sections and report exactly which training-pool cities were used for each knowledge-driven subset.","section":"Section 4.3 and Appendix D.3"},{"comment":"The text in Section 6 is not supported by Table 1 in two places. First, the claimed 'up to 11.9%' improvement for iWildCam does not appear in the table: the largest knowledge-driven improvement over all-data is 0.75 vs 0.72 (Deployment 3), about 4.2% relative or 3 percentage points. Second, the sentence 'Even when the knowledge-driven subsets underperform all training data, as in NuScenes Deployment 2, there exist subsets from other baselines that outperform using all the data' contradicts Table 1, where all-data has MSE 0.021 and every listed subset has MSE at least 0.032. Please correct these claims or identify which table or metric they refer to.","section":"Section 6 and Table 1"},{"comment":"The knowledge-driven subsets are constructed with information that DS3 algorithms will not have, notably the deployment's class list (used to eliminate classes absent from the deployment) and geographic metadata (Section 4.1; Section 4.3; Appendix D). This is disclosed in Section 4, but Section 6's discussion of 'well-chosen subsets' and the abstract's existence framing do not consistently make clear that these are oracle upper bounds rather than selections achievable by an algorithm that only sees the query set. Because the class-list pruning directly encodes part of the evaluation distribution, the reported gains likely conflate two effects: removing irrelevant classes and matching location. Please state explicitly that the knowledge-driven results are oracle bounds computed with privileged information, and consider decomposing the AutoArborist and iWildCam gains into the contribution from class pruning versus location-based selection.","section":"Sections 4.1, 4.3, and 6"}],"minor_comments":[{"comment":"The 'Expert subsets' paragraph for GeoDE appears to be copied from the iWildCam section: it refers to 'the testing pool', 'relevant locations', and a 500km radius, which does not match Section 4.2's description of selecting African subcontinent data for Nigeria, Asian subcontinent data for Indonesia, and indoor/outdoor objects for the other deployments. Please replace this with the correct GeoDE protocol.","section":"Appendix D.2"},{"comment":"The AutoArborist deployment class counts are inconsistent: Section 4.3 and Figure 3 state Calgary has 30 classes, Los Angeles has 175, and Washington DC has 67, while Table 6 lists Calgary as 15, Los Angeles as 30, and Washington DC as 33, even though the listed genus lists agree with the former counts. Please align the header values with the actual class lists.","section":"Table 6 and Figure 3"},{"comment":"The paper mixes relative and absolute gains without definition: AutoArborist's '51.3%' is 0.86 - 0.35 = 0.51 in absolute accuracy (51 percentage points), which is not a relative improvement (about 145% relative); GeoDE's '3.6%' is likewise an absolute difference. Please state consistently whether reported gains are relative or absolute.","section":"Section 6 and abstract"},{"comment":"The citation 'Anonymous (2023)' in the Related Work section is not appropriate for a journal version; please replace it with the actual reference or remove the anonymized citation.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The contradictions in the AutoArborist protocol and the unsupported numbers in Section 6 are extensive enough that I would require a careful revision rather than copy-editing. If the authors correct the protocol descriptions and reconcile the text with Table 1, the benchmark could be a solid contribution; I do not currently see grounds for rejection if the corrected protocol still supports the existence claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The DS3 formalization and the DataS^3 benchmark are a real addition. The problem—pick a training subset for a specific deployment, using only a small query set—is cleanly stated in Eq. 1, and the five real-world datasets with deployment splits fill a gap that DataComp, DataPerf, and SELECT don't cover. The oracle subsets are honestly labeled as using privileged metadata, so they are existence proofs, not blind predictions. On that level the paper works: curated subsets beat all-data training on all twelve deployments, and the effect is largest when label shift is severe. The promise of released code and splits is good.\n\nThe soft spot that matters is the AutoArborist contradiction. Section 4.3 calls Surrey and Calgary in-distribution, with city data in the training pool, and LA/DC out-of-distribution. Appendix D.3 says exactly the opposite. Both sections describe the expert subsets for those deployments differently, which means the flagship 51.3% gain (Surrey: 0.86 vs 0.35) could come from the city's own training images or from a nearest-city proxy. Those are different selection rules, and the paper does not give enough information to tell which one produced Table 1's numbers. This is not a cosmetic typo; it changes the meaning of the headline result and has to be resolved before anyone can reproduce it.\n\nA couple of smaller issues. The baseline suite is thin: no representative coreset methods (GLISTER, GradMatch, random in the main table), so the claim that current methods consistently fail is stronger than the evidence. There are no multiple seeds or error bars, and the gains are not decomposed—how much comes from pruning classes absent from the deployment versus matching location. These are fixable.\n\nThe central hypothesis holds up on the other four datasets, so this is not a case where the whole idea collapses. The benchmark is useful and the paper is worth engaging with. It deserves a serious referee, but the referee should push for a corrected AutoArborist protocol, a richer baseline set, and some variance reporting. My recommendation: send it to peer review, but expect a major revision before it is ready to appear.","headline":"Useful benchmark for deployment-specific data selection, but the AutoArborist ID/OOD contradiction undercuts the flagship number and must be fixed.","tokens_in":26423,"tokens_out":4304,"would_cite":true,"duration_ms":37538,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deployment-specific subsets of a general training pool can beat training on all the data, with gains up to 51.3 percent.","keywords":["dataset subset selection","model specialization","deployment-specific fine-tuning","distribution shift","data curation benchmarks","coresets","unlabeled query set","vision benchmarks"],"falsifier":"Recompute the iWildCam Deployment 1 knowledge-driven subset without the step that removes classes absent from the deployment, keeping only the GPS-proximity filter, and retrain under the same protocol. If accuracy drops to the all-data baseline, the reported gains depend on oracle class-list pruning rather than on selection from the query set.","tokens_in":25344,"feed_emoji":"🎯","tokens_out":8379,"duration_ms":72706,"temperature":0.7,"pith_summary":"The paper formalizes a problem it calls dataset subset selection for specialization (DS3): choose a subset of a general training pool to fine-tune a model for one specific deployment, using only a small query set from that deployment as guidance. It then introduces DataS3, a benchmark with five real-world datasets and twelve deployments, ranging from camera-trap species identification to steering-angle regression. The benchmark's central claim is that a well-chosen, deployment-specific subset of the training pool can outperform training on all available data, with reported gains of up to 51.3 percent accuracy. The paper also reports that existing coreset and data-filtering methods, which assume the training and test distributions match, consistently fail on these deployment tasks, leaving unlabeled-query subset selection as an open problem.","feed_headline":"Small curated subsets beat full training pools on 12 deployments","feed_subtitle":"Expert subsets lift accuracy up to 51.3 percent, and no unlabeled-query selector matches them yet.","key_machinery":"The load-bearing object is the DS3 selection objective (Equation 1): an algorithm receives the training pool $T$ and a query set $Q$ drawn from the deployment distribution, and must return the subset $S\\subset T$ that minimizes the expected deployment loss $\\mathbb{E}_{q\\sim P_Q}\\,L(\\theta(S), q)$. The benchmark wraps this objective in a fixed four-step pipeline: split the deployment pool into query and evaluation sets, select a subset, finetune a fixed model (ResNet50, ViT, or YOLOv8n depending on task), and evaluate on the held-out deployment set. The knowledge-driven subsets act as oracle upper bounds for the existence claim, since they are built from metadata and class lists rather than from the query set itself.","core_discovery":"On the paper's own terms, the discovery is an existence result: for each of the twelve deployments in the benchmark, there is a curated subset of the general training pool whose fine-tuned model beats the model trained on the entire pool. These knowledge-driven subsets are constructed with information that benchmark users are not given, such as GPS proximity between training camera traps and the deployment, matching city, country, or region identity, and the full class list of the deployment evaluation set. The reported margins are up to 3.6% higher accuracy on GeoDE, 11.9% on iWildCam, and 51.3% on AutoArborist, a 0.03 lower mean squared error on NuScenes, and a 0.13 higher mAP50 on FishDetection. A second claim is negative: simple query-set alignment baselines, both image-alignment and nearest-neighbor, do not consistently reach these gains, so the benchmark is offered as an open testbed rather than a solved task.","pith_inferences":["Beyond the paper, separating the two ingredients of the oracle subsets—class-list pruning and geographic proximity—would show which one carries the gains; a method that predicts deployment classes from query images could emulate the first without labels.","The same pipeline could be applied to the future-work domains named in the paper (histopathology, ICU mortality, crop mapping, galaxy classification), providing a quick test of whether subset-existence transfers across data modalities.","The NuScenes result hints that in regression settings, removing irrelevant scenes may matter more than sample count; a direct test would train on random subsets matched for size and compare variance."],"forward_implications":["A model fine-tuned on a deployment-matched subset can beat a model trained on the full general pool, so data selection is a lever for deployment performance independent of architecture.","Subset sizes that achieve these gains average 4–20% of the training pool, so specialization can also cut training compute and storage.","Methods designed for generalization, such as coresets and distribution matching, are not directly transferable to specialization, which is why a dedicated benchmark is needed.","Since labeled query sets are often unavailable in field deployments, progress on unlabeled-query subset selection is the main requirement for practical impact."],"supporting_citations":[{"why":"Supplies the iWildCam camera-trap dataset, its 182 species classes, and the deployment locations used for the iWildCam tasks.","marker":"Beery et al. (2021)"},{"why":"Supplies the AutoArborist multi-city tree dataset from which the four city deployments and class distributions are drawn.","marker":"Beery et al. (2022b)"},{"why":"Supplies the GeoDE object dataset with six world regions that define the Nigeria, Indonesia, indoor, and outdoor deployments.","marker":"Ramaswamy et al. (2023)"},{"why":"Supplies the NuScenes driving dataset with city splits used for the four steering-angle regression deployments.","marker":"Caesar et al. (2020)"},{"why":"Supplies the FishTrack23 underwater video dataset and the geographic subsets used for the four fish-detection deployments.","marker":"Dawkins et al. (2017)"},{"why":"Provides the WILDS-iWildCam test splits whose test-id/test-ood distinction defines which iWildCam deployments are in- or out-of-distribution.","marker":"Koh et al. (2020)"},{"why":"DataComp is the main prior benchmark for data curation and the inspiration for the nearest-neighbor query-alignment baseline; the paper positions DataS3 against it.","marker":"Gadre et al. (2024)"},{"why":"ResNet50 is the fixed full-finetuning architecture used to produce the main accuracy, MSE, and mAP50 comparisons.","marker":"He et al. (2015)"},{"why":"YOLOv8n is the fixed detection model used to evaluate the FishDetection deployments.","marker":"Jocher et al. (2023)"}],"fun_headline_variants":["Curated subsets beat full-data training on all 12 deployments","Privileged curation yields up to 51.3% accuracy boost on specialized tasks","Blind alignment methods fail to match curated subset gains","DataS^3 shows expert subsets outperform full training on 12 deployments","Specialized subset selection: curated beats all-data training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the knowledge-driven subsets are fair upper bounds: they use privileged information (deployment class lists and GPS locations) that users are not given, and the paper's own split descriptions conflict over which AutoArborist deployments count as in-distribution, so the gains may not be approachable by a method that only sees the unlabeled query set.","fun_headline_variants_meta":{"raw":{"variants":["Curated subsets beat full-data training on all 12 deployments","Privileged curation yields up to 51.3% accuracy boost on specialized tasks","Blind alignment methods fail to match curated subset gains","DataS^3 shows expert subsets outperform full training on 12 deployments","Specialized subset selection: curated beats all-data training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3655,"prompt_tokens":1043,"completion_tokens":2612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":2522}},"tokens_in":659,"tokens_out":2612,"duration_ms":18699,"temperature":1.0,"reasoning_tokens":2522,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:07:40.033220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the iWildCam Deployment 1 knowledge-driven subset without the step that removes classes absent from the deployment, keeping only the GPS-proximity filter, and retrain under the same protocol. If accuracy drops to the all-data baseline, the reported gains depend on oracle class-list pruning rather than on selection from the query set.","supporting_citations":[{"cited_title":"The iwildcam 2021 competition dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the iWildCam camera-trap dataset, its 182 species classes, and the deployment locations used for the iWildCam tasks."},{"cited_title":"An open-source platform for underwater image and video analytics","cited_arxiv_id":null,"evidence_quote":"Supplies the FishTrack23 underwater video dataset and the geographic subsets used for the four fish-detection deployments."}],"review_version":1}