{"id":"b6c792ed-eeb9-4980-927b-75f0dc3b12a9","arxiv_id":"2411.09126","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SCAN dynamically prunes and regrows training data during contrastive pre-training, matching full-data accuracy within about 1% on average while using 30-35% less data.","lead":"Pre-training image models on a small, dynamically chosen subset of the data can keep most of the accuracy of full-data training while using 30% less data. The new method, SCAN, repeatedly prunes and regrows the training set and beats static data-selection baselines on CLIP and MoCo models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The per-batch InfoNCE loss criterion in Eq. (2) is never isolated from the bootstrapping schedule; without a dynamic random-candidate control, the 30-35% pruning result may be due to the schedule rather than the loss-extreme selection.","rationale":"I read the paper in good faith. The empirical scope is broad and the authors report code, multiple architectures, two contrastive frameworks, and an honest limitations section that acknowledges the absence of billion-scale validation and the storage cost. Those are real strengths. The load-bearing concern is not about fraud or cherry-picking; it is about whether the proposed selection signal is doing the work attributed to it. The reader identified the same weakness: per-sample in-batch InfoNCE loss is assumed to be a stable, transferable indicator of sample usefulness. My stress-test sharpens this into a specific missing control. The paper's own ablations show that combining redundant and ill-matched samples helps, but they do not compare against a dynamic random-candidate baseline, and the static Random baseline cannot serve that role because it lacks SCAN's schedule and regrowth. Without this control, the abstract's claim that SCAN 'surpasses several baselines with a large performance margin' is not fully tied to the method's distinctive loss-extreme selection. I am not asking for a change in the reader's conditional verdict: the proper response is to keep the paper conditional, with the requested control named as a required revision or a clearly stated limitation. The concrete test I propose is computationally feasible because the released code and schedules already exist; only the candidate-selection line needs to be replaced.","tokens_in":14912,"tokens_out":8875,"duration_ms":87037,"concrete_test":"Using the released code on the ViT-B/32 CC12M+ configuration, run SCAN with exactly one change: at each pruned-data-preparation step, replace the loss-extreme selection of Eq. (2) by drawing the same 2*rho*|Dt| candidates uniformly at random from each batch, while keeping the warm-up, mutation schedule, regrowth, and total epochs identical. Compare zero-shot ImageNet Top-1/Top-5 and CIFAR-100 accuracy against the reported SCAN model. If the random-candidate dynamic control is within 0.5% of SCAN, the loss-based selection is not the active ingredient; if SCAN is more than 1% better across these metrics, the selection criterion is validated. As a secondary check, at the first mutation epoch compute the rank correlation of per-sample losses across three different random batch compositions to quantify the stability of the Eq. (2) signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SCAN matches full-data contrastive pre-training while pruning 30-35% of data depends on the assumption that the per-batch InfoNCE loss extremes selected in Eq. (2) identify samples that are safely removable: low-loss samples as redundant and high-loss samples as ill-matched. This is load-bearing because InfoNCE loss for a sample is computed against randomly sampled negatives in the same batch, so a sample's rank can change substantially across batches and training iterations. The paper never tests the selection criterion in isolation: the 'Random' baseline in Tables 1-3 is a static random 70% subset, not a dynamic random-candidate control that runs SCAN's warm-up, cosine mutation schedule (Eq. 3), regrowth rounds, and epoch count while replacing the loss-extreme rule with uniform random selection. Table 7 compares pruning modes with and without the redundant/ill-matched sets, but it also lacks a random-candidate control. Consequently, the observed gains over static baselines could plausibly arise from the bootstrapping schedule and periodic regrowth alone, with the specific loss-based selection contributing little. This is not an accusation; it is a missing control that is directly needed to support the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCAN, a dynamic bootstrapping dataset pruning method for contrastive pre-training. Instead of selecting a static coreset once, SCAN identifies candidate samples from per-batch InfoNCE losses, keeping the very low-loss samples (treated as redundant) and very high-loss samples (treated as ill-matched), and then prunes a varying number of these candidates through a cosine-annealing mutation schedule with periodic regrowth. The method is evaluated on CLIP pre-training on CC3M+ and CC12M+ and on MoCo pre-training on ImageNet, across 16 total models, with pruning rates of 30–35%. The paper reports less than 1% average downstream degradation relative to full-data training, claims consistent gains over SemDeDup, D-Pruning, Info-Batch, and static random baselines, and shows that the produced static coresets outperform existing static coreset methods.","tokens_in":15178,"tokens_out":10036,"duration_ms":101310,"significance":"If the empirical claims hold, the paper makes a practically useful contribution: it shows that contrastive pre-training can use 30–35% less data on average with a relatively small downstream cost, and that dynamic, schedule-driven pruning can be competitive with or better than static coreset selection. The scope is broad for this line of work: seven CLIP architectures, two MoCo architectures, two CLIP dataset scales, several downstream and out-of-distribution evaluation sets, and comparisons with four baselines. The paper also ships code, which supports reproducibility. The main limitations are statistical and methodological: the headline 'less than 1% average degradation' is not supported by error bars or multiple seeds, and the specific benefit of the loss-extreme selection rule is not isolated from the bootstrapping schedule by a dynamic random-candidate control. These are fixable with additional experiments and more careful wording, rather than being fundamental flaws in the method's design.","major_comments":[{"comment":"The central claim that loss-extreme selection drives the improvement is not isolated from the bootstrapping schedule. The Random baseline in Tables 1–3 is a static 70% subset, and the ablation in Table 7 only compares redundant vs. ill-matched selection with a fixed schedule. No control runs the same warm-up, cosine mutation schedule (Eq. 3), regrowth rounds, and epoch count while replacing Eq. (2) with uniform random candidate selection. Without such a control, the observed gains over static baselines could be attributable to the schedule alone. Please add this control or explicitly weaken the claim.","section":"§3.3.1 Eq. (2), §3.3.2, §4.1 Tables 1–3, §4.4 Table 7"},{"comment":"The abstract's 'less than 1% on average' claim is not backed by per-seed variation or error bars, and the average conceals substantially larger per-metric degradations. For example, in Table 1, ViT-B/32 on CC12M+ drops from 63.40 to 61.55 on ImageNet Top-1 (-1.85 points) and from 31.09 to 29.23 on ImageNet-R (-1.86 points); in Table 3, RN101 on CC3M+ drops from 39.69 to 38.05 on ImageNet-R (-1.64 points). Please report standard deviations over multiple seeds, or explicitly state that results are single runs, and qualify the average-degradation claim accordingly.","section":"Abstract, Tables 1 and 3"},{"comment":"The pruning-ratio bookkeeping is internally inconsistent and prevents exact reproduction. Eq. (3) ranges from 0 to 1 as τcur mod (τcos+1) goes from 0 to τcos, yet the text says 'ρcur increases from 0 to 2ρ'; the candidate set size is 2ρ|Dt| in §3.3.1, §4.1 states a pruning ratio of 30% for CLIP, and Table 8 lists R.v.s.I. ratios (30:10), (20:20), (10:30) that do not match an equal 15%:15% split of a 30% total. Please define ρ, ρcur, and the candidate-set fraction unambiguously and align the text, formulas, and table.","section":"§3.3.2, Eq. (3), and Table 8"}],"minor_comments":[{"comment":"The symbols ≺ρ and ≻ρ are not defined; please state explicitly that they return the indices of the ρ smallest and ρ largest per-sample losses in the batch.","section":"§3.3.1, Eq. (2)"},{"comment":"The phrase 'merge the subset intersection from Lf→g and Lg→f' is ambiguous; please specify whether the final candidate set is the union or the intersection of the two directional selections.","section":"§3.3.1"},{"comment":"The architecture list includes RN50, ViT-S/32, and ViT-S/16, but these do not appear in Tables 1 and 3; please point to the corresponding results in the supplementary material.","section":"§4.1"},{"comment":"The y-axis ranges in Figure 3 are truncated (e.g., FT Acc from 56.00 to 57.00), which visually exaggerates the differences; please use full-scale axes or add error bars.","section":"Figure 3"},{"comment":"There are minor typos: 'doamins' should be 'domains' in Section 2, and 'SACN' should be 'SCAN' in Section 4.2.","section":"§4.2 and §2"},{"comment":"The warm-up threshold Ttd and the mutation epoch τcos are free hyperparameters; please provide their values for each dataset and architecture in a reproducibility table.","section":"§3.3.1 and §3.3.2"}],"recommendation":"major_revision","confidential_remarks":"The main blockers are the missing dynamic random-candidate control and the lack of uncertainty estimates for the headline average-degradation claim. Both are fixable with additional experiments. If the authors add the dynamic random control and report multiple seeds, the empirical claims would be substantially stronger; otherwise the paper currently overstates the role of the loss-extreme selection rule."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper has a real and useful idea — a bootstrapping dataset pruning scheme for contrastive pre-training that removes both low-loss (redundant) and high-loss (ill-matched) samples, with a cosine-annealing mutation schedule that periodically grows the full dataset back and re-selects. The combination is new, even though the components are borrowed. The experimental breadth is a genuine strength: 16 models across CLIP and MoCo, multiple downstream datasets, three evaluation protocols, plus a static coreset export. Code is released. The results show SCAN consistently beats static coreset baselines and is competitive with full-data training while pruning 30-35% of the data.\n\nThat said, there are soft spots. The biggest is that the paper never isolates the loss-based selection from the bootstrapping schedule. The Random baseline is a static 70% subset, not a dynamic control that runs the same warm-up, cosine mutation, and regrowth rounds while picking samples uniformly at random. Without that control, one can't attribute the gains to the loss-extreme rule; the schedule alone might be doing most of the work. This is a testable missing experiment, not a fatal flaw, but it's load-bearing for the paper's central claim.\n\nSecond, there are no error bars or multi-seed results. Several individual metrics drop by 1.5-2 points (e.g., ViT-B/32 on CC12M+ ImageNet-R goes from 31.09 to 29.23). The 'less than 1% on average' claim may be arithmetically true, but without variance we can't tell whether a 2-point drop is noise or a real cost.\n\nThird, Table 7 contains a small overstatement. The text says combining redundant and ill-matched samples 'results in further performance improvement,' but for ViT-S/32 zero-shot Top-5 the combination (33.60) is worse than ill-matched alone (34.12). The union still helps on some metrics, but the claim as written is too strong.\n\nMinor points: 'first comprehensive study' overstates novelty, and the hyperparameters (pruning ratio, mutation epoch, warm-up threshold) are tuned on a subset of models without clear selection details. The limitations section is honest about scale and storage.\n\nWho is this for: people working on data-efficient contrastive pre-training, dataset pruning, or CLIP/MoCo pipelines. They'll find it useful as a baseline and a practical method, but they should replicate the missing control before trusting the attribution.\n\nRecommendation: yes, send to peer review. The work is coherent, reproducible, and the main gap fixable. A revision should add the dynamic random control, report variance across seeds, and correct the Table 7 wording.","headline":"SCAN's bootstrapping schedule and broad experiments are valuable, but the missing dynamic-random control leaves the core attribution unproven.","tokens_in":15692,"tokens_out":4521,"would_cite":true,"duration_ms":93875,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining with 30% less data costs under 1% accuracy","keywords":["contrastive pre-training","data pruning","coreset selection","dynamic sparse training","InfoNCE loss","CLIP","MoCo","data efficiency"],"falsifier":"Record per-sample InfoNCE losses every epoch for a small CLIP model and check rank-order stability between the warm-up epoch and later epochs; if a substantial fraction of samples, say more than 10%, move from the lowest-loss quintile to the highest-loss quintile, or vice versa, the pruning decision would be wrong for those samples. Alternatively, take the pruned set from SCAN at a 30% ratio, add back a random 5% of pruned samples during the final training epochs, and measure whether downstream accuracy rises by more than the reported noise; if it does, those samples were not truly redundant.","tokens_in":14719,"feed_emoji":"✂️","tokens_out":6821,"duration_ms":61829,"temperature":0.7,"pith_summary":"The paper claims that contrastive pre-training can be made substantially more data-efficient by pruning training samples dynamically, rather than selecting a fixed coreset in advance. It introduces SCAN, a bootstrapping method that every few epochs scores each sample by its in-batch InfoNCE loss, removes both the easiest redundant samples and the hardest ill-matched samples under a cosine-annealed schedule, then grows back to the full dataset and repeats. Across 16 pre-trained models spanning CLIP and MoCo, pruning 30-35% of training data keeps downstream accuracy within 1% on average of full-data training and beats static coreset baselines by a large margin. If the claim holds, SCAN offers a cheap way to reduce pre-training compute and carbon footprint without sacrificing model quality.","feed_headline":"Pretraining with 30% less data costs under 1% accuracy","feed_subtitle":"SCAN prunes easy and mismatched samples on a rolling schedule, beating static coreset baselines on CLIP and MoCo.","key_machinery":"The engine of the method is the bootstrapping dataset mutation loop. Every $\\tau_{\\text{cos}}+1$ epochs, the method recomputes candidate pruning data from in-batch InfoNCE losses: the $\\rho$ smallest-loss samples are labeled redundant and the $\\rho$ largest-loss samples are labeled ill-matched, and their union across the two loss directions $L_{f\\to g}$ and $L_{g\\to f}$ forms the candidate pool. Within each round, the actual pruning ratio follows a cosine-annealing curve $\\rho_{\\text{cur}}$ that rises from 0 to $2\\rho$, so the dataset is gradually shrunk to an average ratio $\\rho$ and then restored to full size for the next round. This dynamic schedule is what distinguishes SCAN from static coreset selection, and a warm-up rule—start pruning only after epoch-wise loss differences fall below a threshold—protects early unstable training.","core_discovery":"On its own terms, the paper establishes that dynamic, loss-based dataset bootstrapping during contrastive pre-training can reduce training data by roughly one third while preserving almost all downstream performance. The central discovery is that the per-sample InfoNCE loss computed inside each batch is a usable indicator of sample usefulness across the training trajectory: low-loss samples have been memorized and can be dropped as redundant, high-loss samples are semantically mismatched and can be dropped as ill-matched, and pruning the union of these two extremes under a periodically increasing cosine schedule preserves the information content of the full dataset. The method is validated on seven CLIP architectures on two image-text datasets and two MoCo architectures on ImageNet, for a total of 16 pre-trained models, with an average performance drop below 1% at 30-35% pruning.","pith_inferences":["The in-batch loss signal is cheap enough that SCAN could plausibly scale to billion-sample web datasets, but the paper only tests up to 10.1M pairs, so whether the sub-1% bound survives at web scale is an open question.","The two-sided pruning rule implies a testable claim: samples that are memorized or mismatched are interchangeable across architectures, as the overlap results partially suggest; a direct experiment would check whether a sample pruned by one architecture is ever critical for another.","The cosine-annealed bootstrapping schedule resembles a curriculum over data difficulty; one could test whether the schedule itself, rather than the specific loss cutoff, drives the gains by comparing against fixed-ratio dynamic pruning at the same average rate.","The paper notes that LLM pre-training typically runs for only 1-3 epochs, so the method may transfer more readily to other multi-epoch contrastive objectives, such as sentence embedding training, than to single-pass LLM training."],"forward_implications":["At a 30-35% pruning ratio, SCAN models stay within 1% average downstream performance of models trained on the full dataset, across both CLIP and MoCo.","SCAN consistently outperforms static coreset baselines SemDeDup and D-Pruning, and the dynamic baseline Info-Batch, on zero-shot, linear probing, and fine-tuning evaluations.","Pre-training time drops by roughly 25-30% because the same number of epochs is trained on fewer samples.","The coresets left after SCAN pre-training, especially when intersected across two architectures, outperform existing static coresets when used to pre-train a new model from scratch.","The benefit holds across vision-language (CLIP) and vision-centric (MoCo) frameworks and across ResNet, ViT, and Swin architectures."],"supporting_citations":[{"why":"The dynamic loss-based pruning baseline that SCAN must beat; provides the closest comparison for bootstrapping versus static pruning.","marker":"[48]"},{"why":"A static coreset baseline based on semantic deduplication; comparison shows the advantage of dynamic pruning.","marker":"[1]"},{"why":"A static generalization-influence coreset baseline; appears in all tables as a primary comparison.","marker":"[69]"},{"why":"The vision-language contrastive pre-training objective and backbone for 14 of the 16 models.","marker":"[49]"},{"why":"The vision-centric contrastive framework used for the remaining 2 models.","marker":"[11]"},{"why":"The InfoNCE loss whose per-sample values define the redundant and ill-matched pruning criterion.","marker":"[61]"},{"why":"Supports the claim that low-loss samples have been memorized and are safe to drop.","marker":"[18]"}],"fun_headline_variants":["Dynamic data pruning cuts pretrain data 30% with <1% loss","SCAN prunes easy and mismatched samples on the fly, saving 30% data","Bootstrapped dataset pruning keeps CLIP and MoCo strong with 30% less data","Rolling loss-based pruning trims pretraining data by a third with negligible cost","SCAN's iterative coresets drop 30% of data, lose under 1% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a sample's in-batch InfoNCE loss at pruning time is a reliable and stable signal of its future usefulness—specifically, that low-loss samples stay redundant and high-loss samples stay ill-matched for the rest of training, so neither ever becomes crucial later.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic data pruning cuts pretrain data 30% with <1% loss","SCAN prunes easy and mismatched samples on the fly, saving 30% data","Bootstrapped dataset pruning keeps CLIP and MoCo strong with 30% less data","Rolling loss-based pruning trims pretraining data by a third with negligible cost","SCAN's iterative coresets drop 30% of data, lose under 1% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3555,"prompt_tokens":973,"completion_tokens":2582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2480}},"tokens_in":589,"tokens_out":2582,"duration_ms":17263,"temperature":1.0,"reasoning_tokens":2480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:59:47.155734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record per-sample InfoNCE losses every epoch for a small CLIP model and check rank-order stability between the warm-up epoch and later epochs; if a substantial fraction of samples, say more than 10%, move from the lowest-loss quintile to the highest-loss quintile, or vice versa, the pruning decision would be wrong for those samples. Alternatively, take the pruned set from SCAN at a 30% ratio, add back a random 5% of pruned samples during the final training epochs, and measure whether downstream accuracy rises by more than the reported noise; if it does, those samples were not truly redundant.","supporting_citations":[{"cited_title":"Infobatch: Loss- less training speed up by unbiased dynamic data pruning","cited_arxiv_id":null,"evidence_quote":"The dynamic loss-based pruning baseline that SCAN must beat; provides the closest comparison for bootstrapping versus static pruning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A static coreset baseline based on semantic deduplication; comparison shows the advantage of dynamic pruning."},{"cited_title":"Dataset pruning: Reducing training data by examining generalization influence","cited_arxiv_id":null,"evidence_quote":"A static generalization-influence coreset baseline; appears in all tables as a primary comparison."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"The vision-language contrastive pre-training objective and backbone for 14 of the 16 models."},{"cited_title":"An empirical study of training self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"The vision-centric contrastive framework used for the remaining 2 models."},{"cited_title":"Represen- tation learning with contrastive predictive coding","cited_arxiv_id":null,"evidence_quote":"The InfoNCE loss whose per-sample values define the redundant and ill-matched pruning criterion."}],"review_version":1}