{"id":"58256ab4-23a6-46d9-892f-c92bc9298c65","arxiv_id":"2507.12998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DISSect selects training samples for multimodal contrastive learning by ranking the difference between historical and current model similarity scores, matching full-data performance with 70% fewer samples.","lead":"The paper introduces a data-selection trick for training image-text models like CLIP: it scores each sample by how much the model's predicted image-text similarity has changed since an earlier moment, and trains only on the most informative samples. The result is a training method that reportedly matches full-data performance while using far fewer samples, with no separate oracle model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that the δ = CLIPScore_hist − CLIPScore_curr separation of clean/noisy pairs persists under DISSect's own selective-training loop and on datasets beyond CC3M; this is only shown for vanilla full-data training with an oracle partition.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test pass identifies the same load-bearing assumption: the clean/noisy separation of δ is only demonstrated under vanilla full-data training on CC3M. I sharpen it: DISSect's selection changes the very distribution used to measure the trajectory, so the separation could be a property of full-data training that disappears once selection feedback begins. The proposed test directly measures the AUC of δ under the actual DISSect loop on two held-out datasets. If the test passes, the method's mechanism is supported; if it fails, the empirical wins need to be re-attributed. Secondary issues also reinforce conditionality: the warm-up epoch Tw is not specified in the main text, the derivation of Eq. 4 is omitted, no error bars are reported, and the abstract's '70% fewer iterations' is more precisely 70% fewer samples per iteration rather than fewer iterations. None of these change the verdict, but they should be addressed before the method is used as a drop-in accelerator.","tokens_in":15747,"tokens_out":16390,"duration_ms":211344,"concrete_test":"Train DISSect-Warmup and DISSect-Mome. on CC12M and YFCC15M at r=0.5 using the released code. At epochs 5, 10, and 20, freeze the oracle CLIP-L model, label a stratified sample of 10k pairs as clean/noisy by the median oracle CLIPScore, and compute the ROC-AUC of δ (Eq. 3) on those pairs. Also compute the same AUC for a vanilla full-data run on both datasets. If the AUC under DISSect remains close to or above the vanilla AUC (e.g., >0.6) in both datasets, the trajectory assumption transfers; if AUC drops substantially, the differential signal is an artifact of the selective-training loop and the reported gains need re-attribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the differential δ in Eq. 3. The evidence for it (Figs. 3b, 3c) is produced by training a vanilla model on the full CC3M dataset and then partitioning samples into clean/noisy with an oracle CLIP model. But DISSect changes the training distribution: only the top-r fraction of each batch is used for the contrastive loss (Algorithm 1, line 10), so the history of a sample's CLIPScore is generated by a model that may never have trained on that sample. Unselected samples can remain at their warm-up score, giving δ≈0; selected samples can be repeatedly re-selected, locking the model onto a subset. The paper does not measure whether the clean/noisy gap in δ survives this feedback loop. Fig. 4c/d reports only the final cleanliness of selected samples (65% TP at r=0.3), which is an indirect check and is evaluated against an oracle threshold, not a per-sample δ AUC. If the gap in δ is produced or destroyed by the selection itself, then the method's success on CC12M/YFCC15M could be explained by the reduced effective batch size or by early random sampling rather than by the proposed differential. Supporting theory (Eq. 4) is asserted without derivation, so it does not close this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DISSect, an online batch-level sample selection method for multimodal contrastive learning. Instead of relying on an oracle model or a static loss threshold, DISSect selects, within each batch, the samples with the largest differential delta = CLIPScore_hist - CLIPScore_curr between a historical and the current model prediction of image-text similarity. The method is evaluated on CC3M, CC12M, and YFCC15M with CLIP and BLIP backbones across image-text retrieval, NLVR2 reasoning, and COCO captioning, and it is compared with coreset, online, and noisy-correspondence baselines. The central empirical claim is that DISSect reaches performance comparable to full-data training while using only 30% of the samples, and that it outperforms oracle-based competitors on several settings.","tokens_in":16062,"tokens_out":4923,"duration_ms":59947,"significance":"If the central claim holds, DISSect is a practically valuable and simple contribution: it removes noisy image-text correspondence during CLIP-style pretraining without an oracle model, with reported wall-clock savings, and it is evaluated across multiple datasets, backbones, and downstream tasks. The paper also provides a plausible memorization-based explanation, compares against dual-network noisy-correspondence methods, and releases code. The main weakness is that the load-bearing mechanism is not verified under the method's own selective-training loop, and the theoretical link in Eq. (4) is asserted rather than demonstrated. The significance is therefore real but conditional on the mechanism surviving this check.","major_comments":[{"comment":"The load-bearing premise of §3.4 is that the differential delta separates clean from noisy samples. The evidence for this (Figs. 3b and 3c) is obtained from vanilla full-data training, with clean/noisy labels supplied by an oracle model, and not from the selective-training loop of Algorithm 1. Under DISSect, the training distribution shifts: samples not selected in a batch do not contribute to the loss, so their CLIPScore_curr can remain near the warm-up value, while repeatedly selected samples may be reinforced. The paper does not report the per-sample delta separation under DISSect itself; Fig. 4(c,d) reports only the oracle-threshold true-positive rate of the selected set, which is an indirect measure. I request a direct diagnostic: track delta for oracle-labelled clean and noisy samples during DISSect training (e.g., ROC-AUC or class separation at several epochs) and compare with vanilla training. Without this evidence, the reported acceleration could plausibly be attributed to the reduced effective batch size or to the warm-up random phase rather than to the differential criterion.","section":"§3.3, §3.4, Algorithm 1"},{"comment":"Equation (4) is introduced with the sentence \"we demonstrate in detail in the Appendix,\" but the derivation is not present in the main text, and the displayed formula is not connected to Eq. (3) by a formal argument. As written, the right-hand side averages exp(CLIPScore)^{-1} and is not shown to be larger for noisy than for clean samples, nor is it shown that delta equals or bounds this gradient term. Please supply the full derivation, state the assumptions (notably whether the memorization result of [33] transfers from cross-entropy classification to the InfoNCE loss), and prove or state precisely the claimed relationship between delta and the gradient offset. The theory as it stands does not close the gap between the memorization heuristic and the selection rule.","section":"§3.4, Eq. (4)"},{"comment":"Tables 1, 2, 3, 4, and 5 report single training runs without error bars or significance tests. Several comparisons are close (e.g., Table 1, MS-COCO 30%: DISSect-Warmup IR@10 56.40 versus InfoBatch 54.98; Table 3, NLVR2 dev 73.99 versus SCAN 72.19), and the central claim of \"comparable to full data\" rests on small margins. Please report at least three seeds with means and standard deviations, or otherwise establish that the reported differences are not within run-to-run noise. This is needed for the empirical claim to be load-bearing.","section":"Tables 1-5"}],"minor_comments":[{"comment":"In the Flickr30K (1K Test Set) panel, the Coreset Forgetting row at 30% selection ratio reports TR@10 = 21.30, which is lower than the corresponding TR@1 = 16.46 but implausibly close to the IR@10 value; this appears to be a typographical error and should be corrected.","section":"Table 1"},{"comment":"The caption refers to a \"Bellington\" example; this word is unclear and likely a typo, which should be fixed for readability.","section":"Figure 1 caption"},{"comment":"The phrase \"plays an important direction\" in the abstract and similar grammatical constructions in the introduction should be revised for clarity.","section":"Abstract and Introduction"},{"comment":"The pseudo-code places \"Predict CLIPScore_hist\" inside the warm-up loop, which suggests it is updated only during warm-up; the text should clarify whether the historical score is fixed after warm-up or recomputed, and how this interacts with the temporal-ensembling variant in Eq. (5).","section":"Algorithm 1 and §3.4"}],"recommendation":"major_revision","confidential_remarks":"The practical results are promising and the paper is generally well positioned in the sample-selection literature. The main risk is mechanistic: the selection criterion is validated offline on vanilla training, but the method changes the training distribution, and the theory is deferred. A direct diagnostic of delta separation under DISSect's own loop would substantially increase confidence. The missing error bars are also a concern for the close comparisons. I see this as a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It's a refreshingly simple online sample selection method for CLIP-style pretraining: rank each pair by δ = historical CLIPScore − current CLIPScore, keep the top r per batch. The historical score comes from a warm-up or temporal ensembling, so no oracle model. The main claim is that this accelerates training to full-data quality at 30% selection ratio on CC3M, and it backs that up with experiments on CC12M, YFCC15M, multiple backbones, and downstream tasks. I buy the practical result: the method consistently beats random online selection and most baselines, including oracle-based coreset methods, across datasets and ratios.\n\nWhat's novel is the specific score, the historical-minus-current CLIPScore differential. ELR uses a similar temporal-ensemble signal as a regularizer for noisy labels, not as a selection score, and the online selection methods cited don't use this differential. The paper also does a good job diagnosing why loss-based selection fails in contrastive learning: the distributions overlap too much. Fig. 3's clean/noisy trajectories are a nice empirical observation.\n\nThe soft spots are real but not fatal. The biggest is that the separation in δ is only demonstrated on vanilla full-data training with an oracle partition (Figs. 3b/c). The method changes the training distribution: only top-r pairs get gradients, so unselected pairs may stay near their warm-up score, giving δ≈0, and selected pairs get re-selected. The paper never measures whether the clean/noisy gap in δ survives DISSect's own feedback loop. Fig. 4c/d is an indirect check—final cleanliness at 65% TP is decent but evaluated against an oracle threshold, not per-sample δ AUC. This leaves room for an alternative explanation (e.g., reduced effective batch size or early random sampling). I'd like to see an experiment where DISSect trains, then the trained model's δ is evaluated on an oracle-partitioned sample to see if the gap is still there.\n\nSecond, Eq. 4 is asserted and deferred to the appendix. The derivative sketch is plausible but not a derivation; the paper should either prove it or clearly label it as intuition. Third, no error bars anywhere—single runs for all tables. That's common in this literature but worth flagging. Finally, the abstract's '70% fewer iterations' is loose; at 30% selection ratio you still iterate over all batches, you just compute the loss on a subset. What's actually saved is compute per step, and Table 6 shows ~63% wall-clock savings at that ratio.\n\nThe citation pattern looks fair, and the claimed novelty relative to ELR holds on a careful read.\n\nWho's this for? Anyone working on efficient multimodal pretraining or data selection. It's a practical, simple, well-validated trick. I'd send it to a serious referee expecting minor revisions: add the feedback-loop experiment, report variances, tighten the abstract, and move the derivation or label it correctly. Verdict: deserves peer review.","headline":"Useful oracle-free data selection for multimodal pretraining with a real empirical payoff, but the causal story needs one more experiment to close the loop.","tokens_in":16549,"tokens_out":3535,"would_cite":true,"duration_ms":35791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting image-text pairs by the gap between historical and current similarity scores lets CLIP-style training match full-data performance with 70% fewer samples.","keywords":["sample selection","multimodal contrastive learning","noisy correspondence","CLIPScore","online data selection","training acceleration","memorization effect","vision-language pretraining"],"falsifier":"Measure $\\delta$ on a dataset with known pair labels after the early-learning point; if the $\\delta$ distributions of clean and noisy pairs overlap substantially rather than showing a growing gap, DISSect's ranking would select the wrong samples and its advantage should disappear. A direct test would inject synthetic correspondence noise with controlled timing (e.g., noise that is learned as fast as clean pairs) and check whether DISSect still beats random selection.","tokens_in":15576,"feed_emoji":"⚡","tokens_out":6854,"duration_ms":65770,"temperature":0.7,"pith_summary":"This paper claims that sample selection for CLIP-style contrastive learning can be accelerated without any pretrained oracle model by scoring each image-text pair with the difference between the model's historical and current predicted similarity, the differential-informed score. The key empirical observation is that clean pairs' self-predicted similarity rises early and then eases, while noisy pairs start low and climb later as the model memorizes them, so the historical-minus-current gap separates the two even when the raw scores overlap. Selecting the top fraction of each batch by this gap, the proposed DISSect method matches full-data zero-shot retrieval performance on CC3M at a 30% selection ratio and surpasses full-data training on the noisier YFCC15M dataset at 50% selection. The paper backs the heuristic with a gradient-based argument that links the differential to the memorization effect in noisy-label learning.","feed_headline":"A similarity gap prunes 70% of CLIP training data","feed_subtitle":"Training on the top 30% of pairs ranked by historical-minus-current similarity matches full-data retrieval performance.","key_machinery":"The central object is the sample-wise differential $\\delta = \\mathrm{CLIPScore}_{\\mathrm{hist}} - \\mathrm{CLIPScore}_{\\mathrm{curr}}$, where CLIPScore is a scaled positive cross-modal similarity between image and text embeddings. The historical score is obtained either from a model checkpoint after a warm-up phase or from a momentum-updated temporal ensemble, so no oracle model is needed. The differential acts as a per-sample learning-tendency indicator: clean pairs show near-zero or slightly negative $\\delta$ after the early learning point, while noisy pairs show large positive $\\delta$ as they are memorized. Selecting the top-$kr$ fraction of each batch by $\\delta$ is the selection mechanism that carries the acceleration.","core_discovery":"The central discovery is that the temporal trajectory of a pair's CLIPScore under CLIP-style training encodes its quality: clean correspondences reach high similarity early and then dip slightly, whereas noisy correspondences begin low and rise as the model memorizes them, so their distributions eventually overlap. Computing the differential $\\delta = \\mathrm{CLIPScore}_{\\mathrm{hist}} - \\mathrm{CLIPScore}_{\\mathrm{curr}}$ between a historical model state (obtained by a short warm-up or by temporal ensembling) and the current state turns this trajectory into a per-sample separation score. Ranking each batch by $\\delta$ and training only on the largest-$\\delta$ samples removes noisy correspondence without a reference model, achieving zero-shot retrieval performance comparable to full-data training at 30% selection and a 2.85$\\times$ speedup to reach full-data performance on CC3M.","pith_inferences":["The same historical-minus-current differential could be applied to other self-supervised losses that exhibit memorization, such as unimodal contrastive learning or masked autoencoding, where a similar clean-then-memorize trajectory may hold.","Because DISSect is oracle-free, it could be composed with offline coreset pruning in a two-stage pipeline: first use DISSect during a short warm-up to filter the corpus, then train on the filtered set.","The trajectory assumption could be tested as a diagnostic: datasets or training schedules that do not show the clean-peak-then-ease pattern would be poor candidates for DISSect, potentially guiding when to use selection at all.","The reported sensitivity of final performance to selection ratio (best at 60%, stable above 50%) suggests there is an optimal budget that trades diversity against noise removal; finding it adaptively during training is a natural next step."],"forward_implications":["At a 30% selection ratio, DISSect reaches zero-shot retrieval performance on MS-COCO and Flickr30K comparable to full-data training on CC3M, meaning roughly 70% of iterations can be skipped without an oracle.","On the noisier YFCC15M dataset at 50% selection, DISSect outperforms full-data training, suggesting that removing noisy correspondence improves quality, not just efficiency.","DISSect works with different backbones: when applied only to the image-text alignment module, it improves both CLIP/ResNet101 and BLIP/ViT-B over random sampling and SCAN.","A single-network DISSect beats dual-network noisy-correspondence methods NCR and GSC in downstream retrieval while avoiding their extra compute cost.","The temporal-ensembling version requires no warm-up epoch and is insensitive to the momentum hyperparameter, making the method plug-and-play."],"supporting_citations":[{"why":"Defines CLIPScore, the per-pair similarity measure whose historical-minus-current difference forms the selection score.","marker":"[15]"},{"why":"Supplies the memorization-effect theory that grounds the claim that noisy samples are learned late and produce the observed differential.","marker":"[33]"},{"why":"Provides the temporal ensembling update used to estimate the historical score without a warm-up epoch.","marker":"[24]"},{"why":"Defines the InfoNCE contrastive loss whose sample-wise gradient behavior motivates the differential in the theoretical analysis.","marker":"[38]"},{"why":"Provides the CLIP-style contrastive learning recipe and training settings on which all experiments are built.","marker":"[7]"}],"fun_headline_variants":["Similarity shift between model states prunes 70% of CLIP data","Differential-based selection cuts CLIP pretraining data to 30%","Rank pairs by historical-minus-current similarity, train top 30%","Model memory gap flags noisy pairs, 2.85x faster CLIP training","No oracle, no extra data: trajectory gap selects 30% for full performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the temporal trajectory pattern observed on CC3M generalizes: across datasets, noise types, and architectures, clean pairs' self-predicted similarity peaks early and then eases while noisy pairs start low and rise late, so that the historical-minus-current differential cleanly separates them.","fun_headline_variants_meta":{"raw":{"variants":["Similarity shift between model states prunes 70% of CLIP data","Differential-based selection cuts CLIP pretraining data to 30%","Rank pairs by historical-minus-current similarity, train top 30%","Model memory gap flags noisy pairs, 2.85x faster CLIP training","No oracle, no extra data: trajectory gap selects 30% for full performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3330,"prompt_tokens":929,"completion_tokens":2401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2301}},"tokens_in":545,"tokens_out":2401,"duration_ms":19378,"temperature":1.0,"reasoning_tokens":2301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:32:54.488055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure $\\delta$ on a dataset with known pair labels after the early-learning point; if the $\\delta$ distributions of clean and noisy pairs overlap substantially rather than showing a growing gap, DISSect's ranking would select the wrong samples and its advantage should disappear. A direct test would inject synthetic correspondence noise with controlled timing (e.g., noise that is learned as fast as clean pairs) and check whether DISSect still beats random selection.","supporting_citations":[{"cited_title":"Early-learning regularization pre- vents memorization of noisy labels","cited_arxiv_id":null,"evidence_quote":"Supplies the memorization-effect theory that grounds the claim that noisy samples are learned late and produce the observed differential."},{"cited_title":"Reproducible scal- ing laws for contrastive language-image learning","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP-style contrastive learning recipe and training settings on which all experiments are built."}],"review_version":1}