{"id":"b7120e85-843c-48a6-95ca-b9b821614d46","arxiv_id":"2506.01586","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MDW distills noisy image-text data into small clean synthetic sets using learnable soft matching probabilities, Grad-CAM guided pixel weighting, and a noise-tolerant negative match loss.","lead":"This paper introduces MDW, a method for compressing large noisy image-text datasets into tiny clean synthetic datasets by learning soft sample-level correspondences and a negative-match loss. If it works as reported, it could make multimodal model training much cheaper and more robust to web-crawled mismatched pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1 is proved for label flips in a fully annotated N×N matching matrix, but MDW's target and its own synthetic noise are diagonal content mismatches whose observed labels stay positive; L_n's gradient behavior is not noise-tolerant in that setting, and no proof is supplied.","rationale":"The empirical core is substantial: Tables 1, 2, 4, 5, and 6 show consistent gains over strong baselines, including real CC104K, cross-architecture evaluation, and component ablations. That supports the practical claim that MDW works on the tested benchmarks. The weak point is the coupling between theory and problem: Theorem 3.1's assumptions are not those of the PMP setting, and L_n's implicit diagonal-positive behavior is the opposite of robust to content mismatch. The appendix's consensus-driven sample selection may mitigate the harm from noisy diagonals, but the theorem is not the source of that mitigation. This concern does not refute the empirical results; it refutes the 'certifiable' part of the central claim and should be settled by the proposed experiment before acceptance. The reader's concern about uniform versus structured noise is related, but the sharper issue is that even the paper's own synthetic content-mismatch injection is not covered by the stated theorem.","tokens_in":20978,"tokens_out":13604,"duration_ms":156938,"concrete_test":"Re-run the Table 1 experiment at η=30% on Flickr30K with the exact shuffle-based PMP injection (observed labels remain positive). Train an expert for 10 epochs using only L_n (Eq. 5) and record the mean diagonal similarity h(V_i,T_i) on the known mismatched subset at epochs 1, 5, and 10. If this mean rises over training or fails to stay clearly below a matched-control baseline trained with L_c on the same data, L_n is reinforcing PMPs rather than tolerating them, showing that Theorem 3.1's label-flip model does not cover the claimed setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical guarantee is Theorem 3.1 in Section 3.4. It assumes uniform corruption of matching labels: y_ii flips 1 to 0 with probability η, and y_ij flips 0 to 1 with probability η/(N−1). This is a label-noise model on a fully observed N×N label matrix. The PMP problem MDW claims to solve, and the noise actually injected in Table 1 ('randomly shuffling images for specific noise ratios η'), is different: a fraction of diagonal image-text pairs are semantically unrelated, while the observed positive label for the paired caption is unchanged. Web data such as CC3M similarly provides only one paired caption per image, not an N×N label matrix. The loss L_n in Eq. 5 sums over j≠i only; minimizing it drives every p_ii toward 1, because reducing all off-diagonal probabilities makes the diagonal the unique remaining mass. Under label-flip noise this can be harmless or helpful, since L_n ignores the flipped labels. Under diagonal content mismatch, the same mechanism supplies positive gradients for mismatched pairs, so L_n is not noise-tolerant for the actual PMP setting. Algorithm 1 applies L_n to 'negative matches across all samples', so noisy diagonals are not excluded from this effect. Moreover, no proof of Theorem 3.1 appears in the main text or any appendix, so the scope of the statement cannot be checked. The empirical gains may still be real, but they are not certified by the theorem as presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MDW, a framework for distilling noisy multi-modal image-text datasets into compact clean synthetic datasets. The method combines fine-grained correspondence-enhanced distillation (learnable soft matching probabilities and Grad-CAM-based adaptive weighting) with dual-track collaborative learning (DTCL), which trains on clean positive pairs identified by a consensus-driven filter and on negative matches across all samples via a non-correspondence loss L_n. The authors state a theorem claiming L_n is noise tolerant under uniform label corruption and report extensive experiments on Flickr30K, COCO, and CC104K with synthetic and real noise, showing consistent improvements over prior multi-modal distillation methods. The paper also includes scalability, architecture-transfer, and efficiency analyses.","tokens_in":21319,"tokens_out":3197,"duration_ms":36474,"significance":"If the results hold, MDW would be the first framework to address partially mismatched pairs in multi-modal dataset distillation, an important practical problem given the prevalence of noisy web-crawled data. The empirical study is extensive: multiple datasets, noise ratios, distilled sizes, ablations, cross-architecture evaluation, and a real-world noisy benchmark (CC104K). The reported gains over strong baselines are large and consistent, and the efficiency analysis is useful. However, the theoretical guarantee that underpins the 'certifiable noise tolerance' claim is not proven in the manuscript, and its stated noise model does not match the actual PMP setting studied in the experiments. The empirical comparisons also lack error bars, so the statistical significance of the margins is not established.","major_comments":[{"comment":"The theorem is stated without proof in the main text or any appendix. Since the abstract, introduction, and contribution list claim 'certifiable noise tolerance' and the theorem is the only formal support for this claim, a complete proof (or a precise pointer to an appendix containing one) is required. In addition, the theorem's noise model assumes label flips on a fully observed N×N matrix (y_ii flips from 1 to 0, y_ij flips from 0 to 1), whereas the PMP setting studied in the paper is a diagonal content mismatch: the observed positive label for a mismatched image-text pair remains 1, and the synthetic noise in Table 1 is injected by shuffling images. The theorem as stated does not cover this setting, so the 'certifiable' claim is currently unsupported for the actual problem the paper targets.","section":"Section 3.4, Theorem 3.1"},{"comment":"The non-correspondence loss L_n = -Σ_i Σ_{j≠i} (log(1-p_i2t_ij) + log(1-p_t2i_ij)) only penalizes off-diagonal entries. Under the diagonal-mismatch noise model, a mismatched pair (V_i, T_i) still has y_ii=1 in the observed data, and L_n imposes no penalty on p_ii; indeed, making all off-diagonal probabilities small can drive the diagonal to dominate, which may strengthen the model's confidence on mismatched diagonals rather than suppress it. The paper needs a gradient-level analysis of L_n under content mismatch (or an explicit argument for why the labeled-diagonal assumption is unnecessary), because the current text only justifies L_n under the label-flip model.","section":"Section 3.4, Eq. (5) and Algorithm 2"},{"comment":"No variance information is reported. Every table gives point estimates without standard deviations or the number of random seeds, so the claimed improvements (15–30% in R_sum) cannot be assessed for statistical significance. Please report mean ± std over at least three, preferably five, random seeds for the main comparisons in Tables 1 and 2, and ideally also for the ablation in Table 3.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'In essence, Ln ensures model convergence to a solution that deviates from the optimal one obtained with clean matching labels by a bounded margin' introduces a 'bounded margin' that is not formally defined anywhere; either define it in the theorem statement or remove it.","section":"Section 3.4 after Theorem 3.1"},{"comment":"The text states that the consensus-driven sample selection is 'training-free and parameter-independent,' but the clean subset depends on the threshold δ and on the fitted Beta Mixture Model parameters; δ is itself a tuned hyperparameter in Table 12. Please qualify this claim.","section":"Appendix A.1, Eq. (6)"},{"comment":"The global-level partition fits a Beta Mixture Model to cosine similarities, but cosine similarity can be negative, while the beta distribution has support [0,1]. Please describe how the similarities are transformed or truncated before fitting, or justify the use of a beta distribution.","section":"Appendix A.1"},{"comment":"The exponential moving average update for A_i is mentioned but its schedule (momentum, update frequency) is not specified; please give the exact update rule in the main text or Appendix F.","section":"Section 3.3, Eq. (4)"},{"comment":"The caption says 'Model on original dataset training achieves ...' but it is not stated whether this refers to clean original data or noisy original data; clarifying this would help interpret the upper-bound comparisons.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is substantial and likely real, but the theoretical certification is currently a load-bearing overclaim: the proof is missing and the stated noise model does not match the PMP setting. I would ask the authors to either supply a proof for the actual setting or explicitly limit the certification claim to the label-flip case. The lack of error bars is also a concern for a paper that emphasizes margins of 15–30%."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a real contribution to multimodal dataset distillation, the first to tackle noisy (partially mismatched) data head-on. The empirical gains are substantial and consistent: on CC104K with real web noise, 100 distilled pairs give R_sum 87.1 vs 67.9 for LoRS, and they also beat the filtered baselines LoRS-C and LoRS+. The building blocks are known—trajectory matching, soft labels, Grad-CAM weighting, negative mining—but the combination is new, and the ablations show the soft matching probability matrix and the correspondence-enhanced optimization each help.\n\nThe main soft spot is the theory. Theorem 3.1 states that Ln is noise tolerant under uniform corrupted matching labels in an N×N label matrix. That is a label-flip model: positive diagonal labels flip to negative, and off-diagonal negatives flip to positive. But the actual PMP setting, and their own synthetic noise injection, is different: a diagonal pair is semantically mismatched while the observed positive label stays 1. Under that model, Ln—which only suppresses off-diagonal probabilities—does not address the noisy diagonal at all. Moreover, no proof of Theorem 3.1 appears anywhere in the main text or appendix, so the scope of the claim cannot be checked. The method's practical robustness likely comes from the memorization-based clean subset selection that filters noisy diagonals out of Lc, plus negative learning; the theorem as stated does not certify that. The authors should either prove the theorem for the actual noise model or reframe the claim.\n\nOther gaps are less severe but real: no error bars or seed counts, no code release, and the CEO hyperparameters beta and k are absent from the hyperparameter table. None of this undermines the empirical result, which holds across noise ratios, distilled sizes, and architectures. I'd send this to peer review; the theory section needs a rewrite, but the core method and evaluation are worth referee time.","headline":"A genuinely new and empirically strong method for distilling noisy multimodal data, but the theoretical claim is proved for a different noise model and the proof is missing.","tokens_in":21833,"tokens_out":1874,"would_cite":true,"duration_ms":21372,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims noisy web-scale image-text data can be distilled into tiny clean sets with provable tolerance to uniform label noise.","keywords":["multi-modal dataset distillation","noisy correspondence","partially mismatched pairs","image-text retrieval","trajectory matching","noise-tolerant loss","learnable matching probability","dual-track collaborative learning"],"falsifier":"Run MDW on a dataset where mismatches are semantically related rather than randomly shuffled (e.g., each noisy caption describes a scene similar to the image) and compare R@1 with the random-shuffle baseline; a large performance drop would show the uniform independent noise model in Theorem 3.1 misses the noise that matters.","tokens_in":20804,"feed_emoji":"⚗️","tokens_out":7477,"duration_ms":71313,"temperature":0.7,"pith_summary":"MDW claims to be the first method that distills large, web-crawled image-text datasets containing partially mismatched pairs (PMPs) into tiny clean synthetic datasets for model training. It replaces hard one-to-one matching labels with a learnable soft matching probability matrix, and adds a dual-track collaborative learning loss on negative matches that the paper proves is noise tolerant against uniform corrupted matching labels (Theorem 3.1). The paper reports that datasets distilled by MDW improve retrieval performance of models trained from scratch by over 15% compared with prior distillation methods, including on a real-world noisy subset of CC104K. If correct, this means practitioners could skip expensive manual data cleaning and still train competitive retrieval models on datasets hundreds to thousands of times smaller.","feed_headline":"Noisy web data distilled into tiny clean sets, up 15%","feed_subtitle":"A learnable soft matching matrix plus a noise-tolerant negative loss beats prior distillation methods on image-text retrieval.","key_machinery":"The load-bearing objects are the learnable soft matching probability matrix $\\tilde{P}\\in\\mathbb{R}^{M\\times M}$, which replaces hard binary match labels and lets each distilled image relate to several texts, and the dual-track collaborative learning (DTCL) loss $L_n$ built from negative matches across all samples. The paper proves Theorem 3.1: for noise ratio $\\eta < (N-1)/N$, $L_n$ is noise tolerant against uniform corrupted matching labels in instance-level image-text matching, meaning its expected gradient direction stays close to the clean-label optimum. The third component is correspondence-enhanced optimization (CEO), which uses Grad-CAM activation maps of the expert visual encoder to compute per-pixel weights that amplify updates on correspondence-discriminative regions of distilled images. Together they carry the argument that distilled data can encode dense correspondence knowledge without being poisoned by PMPs.","core_discovery":"On the paper's own terms, the central discovery is that noisy multi-modal dataset distillation can be made robust by treating non-correspondence as a first-class supervision signal. The proposed MDW framework jointly optimizes synthesized images, text embeddings, and a soft matching probability matrix $\\tilde{P}$ to capture fine-grained sample-level correspondences, and uses a negative-match loss $L_n = -\\sum_i \\sum_{j\\neq i}(\\log(1-p^{i2t}_{ij}) + \\log(1-p^{t2i}_{ij}))$ whose noise tolerance is certified by Theorem 3.1 under uniform label corruption. This lets the expert model learn reliable cross-modal priors from noisy data without discarding most of the dataset, and then the distilled data are further refined by adaptively boosting updates in correspondence-discriminative image regions identified via Grad-CAM-style activation maps. The empirical claim is that models trained from scratch on the distilled sets surpass prior multi-modal distillation methods by more than 15% on standard retrieval benchmarks, with the gap widening as noise increases.","pith_inferences":["Not shown in the paper: the proof of Theorem 3.1 covers uniform, independent label corruption; real web noise is structured and correlated, so the certificate would likely need a stronger noise model to extend to arbitrary PMPs.","The CEO weighting mechanism effectively edits distilled images to amplify task-relevant patterns, which suggests distilled images are not merely compressed data but actively constructed training examples; this could be tested by measuring whether downstream classifiers trained on them inherit the amplified cues.","The soft matching probability matrix could be reused as a pseudo-label prior for other multi-modal tasks such as captioning or visual question answering, though the paper only evaluates retrieval.","The sample-selection stage relies on the memorization effect of neural networks; at much larger scales or with very high noise ratios, that signal may weaken, and MDW's robustness would then depend on how much $L_n$ alone can compensate."],"forward_implications":["A 100-pair distilled set can replace training sets of 29k to 113k pairs for image-text retrieval, with MDW reporting higher R@1/R@5/R@10 than prior distillation methods at the same size.","Using negative matches across all samples raises usable supervision from roughly $0.49N^2$ pairs after filtering to about $N^2 - 0.3N$ pairs, with no extra memory cost because the similarity matrix is already computed.","The distilled datasets transfer across different visual architectures; models trained on MDW-distilled data retain a performance advantage over baselines when evaluated with ResNet or RegNet encoders.","MDW scales with distilled set size: at 2,000 distilled pairs (under 1% of the original data) it reaches about 74% of the full-data retrieval performance on noisy Flickr30K.","The noise-tolerant negative loss also improves the clean-subset selection accuracy used for positive-match learning, reaching 99.7% on Flickr30K with 30% noise."],"supporting_citations":[{"why":"LoRS, the low-rank similarity-mining distillation baseline that MDW compares against and extends.","marker":"[43]"},{"why":"VL-Distill, the prior multi-modal distillation method whose expert-student trajectory matching MDW builds on.","marker":"[40]"},{"why":"Dataset distillation by matching training trajectories; supplies the core trajectory matching objective.","marker":"[4]"},{"why":"Learning with noisy correspondence; provides the memorization-effect-based PMP identification used in sample selection.","marker":"[16]"},{"why":"Grad-CAM; supplies the activation maps used to locate correspondence-discriminative regions in CEO.","marker":"[33]"},{"why":"Robust loss functions under label noise; provides the $L_{mae}$ baseline that $L_n$ is compared with for noise tolerance.","marker":"[12]"},{"why":"Conceptual Captions; source of the real-world noisy CC104K data used in experiments.","marker":"[35]"}],"fun_headline_variants":["Noise-robust multi-modal distillation beats prior by 15%","MDW: first framework to distill noisy multi-modal data","Clean sets from noisy web data, 15% retrieval gain","Learn to ignore noise in multi-modal distillation, +15%","From noise to clean: robust dataset distillation, +15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of noise tolerance assumes mismatched pairs are created by randomly flipping labels independently with the same probability; real web mismatches are correlated and structured, so if that uniformity fails the certified guarantee does not apply.","fun_headline_variants_meta":{"raw":{"variants":["Noise-robust multi-modal distillation beats prior by 15%","MDW: first framework to distill noisy multi-modal data","Clean sets from noisy web data, 15% retrieval gain","Learn to ignore noise in multi-modal distillation, +15%","From noise to clean: robust dataset distillation, +15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000426,"raw_usage":{"total_tokens":2185,"prompt_tokens":950,"completion_tokens":1235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1148}},"tokens_in":566,"tokens_out":1235,"duration_ms":9907,"temperature":1.0,"reasoning_tokens":1148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:37:29.421476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MDW on a dataset where mismatches are semantically related rather than randomly shuffled (e.g., each noisy caption describes a scene similar to the image) and compare R@1 with the random-shuffle baseline; a large performance drop would show the uniform independent noise model in Theorem 3.1 misses the noise that matters.","supporting_citations":[{"cited_title":"Low-rank similarity mining for multimodal dataset distillation","cited_arxiv_id":null,"evidence_quote":"LoRS, the low-rank similarity-mining distillation baseline that MDW compares against and extends."},{"cited_title":"Dataset distillation by matching training trajectories","cited_arxiv_id":null,"evidence_quote":"Dataset distillation by matching training trajectories; supplies the core trajectory matching objective."},{"cited_title":"Learning with noisy correspondence for cross-modal matching.Advances in Neural Information Processing Systems, 34:29406–29419, 2021","cited_arxiv_id":null,"evidence_quote":"Learning with noisy correspondence; provides the memorization-effect-based PMP identification used in sample selection."}],"review_version":1}