{"id":"6457ea22-a1f9-4484-9b4e-202d333756b9","arxiv_id":"2611.16418","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"MDA-DD uses a dynamic queue of models at different training stages and new matching objectives to synthesize compact image-text datasets that outperform prior methods by large margins.","lead":"This paper introduces a method, MDA-DD, for compressing large image-text datasets by generating small synthetic datasets that train vision-language models more effectively. The method uses multiple models at different training stages to supervise the synthetic data, achieving strong retrieval performance with very few synthetic pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader identified the weakest assumption as the theoretical analysis assuming independent model evolution and Lipschitz continuity. This is a valid point about the idealization, but the paper explicitly acknowledges these limitations (b331–b338) and does not present the theory as a proof of convergence—it is offered as intuition. The empirical evidence is the real argument: every component is ablated, baselines are fairly re-implemented, and results are consistent across datasets and budgets. No hidden confound or inconsistency was found. Thus the verdict should remain ACCEPT.","tokens_in":24586,"tokens_out":1882,"duration_ms":15624,"concrete_test":"Reproduce the main Flickr30K 100-pair result (IR@1=24.4, TR@1=34.4) using the provided code and default hyperparameters (Table 8) on the same hardware; verify that the mean over 5 runs is within ±1 percentage point of the reported value. If reproducibility holds, the central claim is solid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MDA-DD outperforms prior methods across budgets, including outperforming 500-pair baselines with only 100 pairs—is supported by extensive empirical evidence: controlled re-implementations of baselines, ablations that isolate each component (Section 4.4, Appendix F), cross-architecture generalization (Table 10), and a third benchmark (LLaVA-CC3M, Table 7). The weakest assumption in the theoretical appendix (independent model evolution, Lipschitz continuity) is explicitly stated (sentences b305–b338) and the paper acknowledges that the concentration bounds hold only for a fixed synthetic dataset and independent trajectories, not during joint optimization (b331, b334). The empirical ablations (queue size K, lifetime Tmax, component contributions) are consistent with the motivation even if the theoretical idealization is imperfect. Because the core empirical claim does not depend on the formal correctness of Appendix C, no load-bearing concern emerges that would weaken the central results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes MDA-DD, a method for multimodal dataset distillation that extends feature-matching across model states using a dynamic staggered queue of models at different training stages. Two additional contributions are asymmetric dual projection (ADP) for complementary alignment in visual and linguistic spaces, and an uncentered second-order cross-modal moment (SOCM) objective that retains mean-related information. Experiments on Flickr30K and MS-COCO show that MDA-DD consistently outperforms existing methods across multiple synthetic-data budgets (100, 200, 500, and 5000 pairs), including surpassing all 500-pair baselines with only 100 pairs on both datasets across all six retrieval metrics. The method also generalizes to unseen image encoder architectures and to a larger web-sourced corpus (LLaVA-CC3M).","tokens_in":24923,"tokens_out":1796,"duration_ms":15201,"significance":"If the reported results hold, MDA-DD represents a substantial advance in multimodal dataset distillation. The key finding—that 100 synthetic pairs can outperform 500 pairs from prior methods—implies a 5× reduction in synthetic data requirements, which is practically significant for reducing storage and training costs in vision-language pre-training. The dynamic model queue is conceptually novel for dataset distillation and addresses a known limitation of model-state overfitting. The empirical evaluation is extensive, covering multiple budgets, two standard benchmarks, cross-architecture generalization, and a third benchmark, with controlled re-implementations of baselines and ablations that isolate each component. The paper is clearly written and the code is provided, supporting reproducibility.","major_comments":[{"comment":"The SOCM component shows negligible or slightly negative marginal gain when added to DQ+ADP: at 100 pairs mR change is +0.10, at 200 pairs -0.23, at 500 pairs -0.13. The paper reports this honestly but does not discuss whether the uncentered moment provides any benefit beyond computational simplicity. Since SOCM is listed as a main contribution, the evidence for its effectiveness beyond computational savings (1.8% faster moment matching according to §4.4) is weak. The authors should either present additional experimental support (e.g., ablations without DQ or ADP) or downgrade SOCM from a core contribution to a minor technical variant and adjust the claims accordingly.","section":"§4.4, Figure 4 and §F.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The standard deviations reported for MDA-DD on ResNet+BERT and RegNet+BERT are considerably larger than those of CovMatch (e.g., IR@1 = 14.5±2.2 vs. 10.6±0.8 for ResNet). While the mean improvements are clear, the higher variance suggests that either the method is less stable across runs or the 5 runs are insufficient to reliably estimate the mean. The authors should comment on this variance and, if possible, report more runs or provide a statistical significance test (e.g., paired t-test against CovMatch over the same seeds) to confirm that the gains are robust.","section":"Table 10 (Cross-architecture)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The theoretical analysis in Appendix C.3 makes several explicit assumptions (bounded features, Lipschitz continuity, independent model trajectories) and explicitly states that the bounds hold only for a fixed synthetic dataset and not during joint optimization (§C.3.3, b331, b334). The authors are transparent about these limitations. However, the main text (§3.1) introduces the queue motivation with a generalization gap ∆state (Eq. 5) and the distributional objective (Eq. 6) without qualifying that the subsequent approximation (Eq. 7) relies on these unverified assumptions. The connection between the theoretical motivation and the actual algorithm is therefore partly aspirational. The authors should add a brief caution in §3.1 that the analysis in Appendix C assumes independent trajectories and a fixed synthetic set, and that the empirical results are ultimately the primary support for the queue's effectiveness.","section":"Appendix C.3","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The caption mentions 'The red dashed circle highlights a subset of CovMatch text embeddings' but the figure appears to be in grayscale in the arXiv version; please ensure the color reference is unambiguous or use a different indicator.","section":"Figure 2 caption","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The header reads 'Preprint.' on the first page. This is acceptable but typically not included in a journal submission. The authors should remove it for the final version.","section":"Page 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption states 'The real-data baseline uses equally sized subsets sampled from the original training data.' Please specify how these subsets are selected (randomly?). Also, report standard deviations if available.","section":"Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The bar colors in Figure 4 are not explicitly defined in the caption; please add a legend or textual description identifying each configuration (e.g., 'DQ enabled' vs. 'DQ disabled').","section":"§4.4, Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The ablation reproduces CovMatch under the same hardware and optimizer settings, which is good practice. Please also report the CovMatch results from this controlled setting as a single row in Table 1 for transparency, even if they match the original numbers.","section":"Appendix F.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The limitations paragraph notes the dependence on CLIP-style dual-encoder architectures. Consider also noting that the method currently requires separate queues for each modality? (It does not, but clarifying may be helpful.)","section":"§5, Limitations","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Some references are to 'to appear' or 'in preparation' (e.g., [22], [54]). Please update to published versions where available, as required by journal policy.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper reports a meaningful advance in multimodal dataset distillation. The main idea—using a dynamic staggered queue of online models at different training stages to supervise synthetic data—is well-motivated and the empirical payoff is clear: with 100 synthetic pairs, they beat 500-pair baselines on both Flickr30K and MS-COCO across all six retrieval metrics. The experiments are thorough: multiple budgets, cross-architecture transfer, ablations that isolate each component, and a third benchmark on noisy web data (LLaVA-CC3M). The code is released. The central claim is solid.\n\nThe dynamic queue (DQ) is the heavy lifter, as the ablations show. Asymmetric dual projection (ADP) gives a genuine secondary gain. The SOCM term, meanwhile, is effectively neutral: at 500 pairs it changes mean Recall by –0.13 points, and the timing improvement is 1.8% on the moment-matching step. The authors are honest about this, and it is a minor soft spot—not a flaw, but worth noting if you assign this paper.\n\nThe theoretical appendix (C.3) is labeled as an analysis of an idealized setting, with independence and Lipschitz assumptions that are acknowledged not to hold during joint optimization. I would not lean on it for a decision; the empirical ablations stand on their own.\n\nWho this is for: anyone working on dataset distillation or efficient vision-language pre-training. The paper is written clearly, the method is simple enough to implement, and the improvements are practically relevant at low data budgets.\n\nVerdict: This deserves a serious referee. I would accept it with minor comments—mostly to note the SOCM contribution is marginal and to suggest the theoretical section be trimmed or caveated more explicitly. It is a solid, reproducible piece of work.","headline":"Solid empirical paper on multimodal dataset distillation; the dynamic queue is the real contribution, and the results hold up.","tokens_in":25378,"tokens_out":1020,"duration_ms":9246},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Model-distribution-aware distillation achieves state-of-the-art multimodal dataset compression with 5× data efficiency","keywords":["dataset distillation","multimodal learning","vision-language retrieval","contrastive learning","distribution matching","model distribution","asymmetric dual projection","second-order moment"],"falsifier":"If a new multimodal distillation method using 100 synthetic pairs fails to exceed the best 500-pair baseline on Flickr30K image-to-text recall@1, then the central claim of 5× data efficiency would be refuted.","tokens_in":26115,"feed_emoji":"🧠","tokens_out":3283,"duration_ms":25464,"temperature":0.7,"pith_summary":"Multimodal dataset distillation aims to compress vision-language training data by synthesizing compact image-text pairs. Existing feature-matching methods cause synthetic data to specialize to the specific model states encountered during distillation, limiting generalization. This paper introduces MDA-DD, which maintains a dynamic queue of models at different training stages to provide diverse supervision. It also uses asymmetric dual projection to align both visual and linguistic spaces and an uncentered second-order moment matching that retains mean information. On Flickr30K and MS-COCO, 100 synthetic pairs trained with MDA-DD achieve higher retrieval accuracy than 500 pairs from any prior method, demonstrating a fivefold reduction in synthetic data need.","feed_headline":"100 synthetic pairs outperform 500 in cross-modal retrieval","feed_subtitle":"New model-distribution-aware method tops all prior work on Flickr30K and MS-COCO with only 0.3% of training data","key_machinery":"The central mechanism is the dynamic staggered model queue, which concurrently maintains up to K online models at different training stages (ages spaced by Δt steps), providing supervision from early, intermediate, and late training states. The asymmetric dual projection (ADP) replaces the single shared-space projection with two cross-dimensional matrices that project image features into text space and text features into image space, enabling complementary alignment. The second-order cross-modal moment (SOCM) replaces centered covariance with the raw moment $M_\\theta(D) = \\frac{1}{n}\\sum e_i^v (e_i^l)^\\top$, which retains mean-related information without centering.","core_discovery":"The paper claims that the key to effective multimodal dataset distillation lies in exposing synthetic data to a distribution of model states rather than a single training trajectory. By maintaining a staggered queue of up to eight online models at different ages, each with independent projection heads and training randomness, the synthetic pairs are optimized to match feature statistics across multiple evolving feature geometries. Combined with asymmetric dual projection that computes contrastive losses in both visual and linguistic spaces, and a second-order cross-modal moment that preserves the outer product of modality means, the method achieves state-of-the-art retrieval performance at extremely low synthetic budgets.","pith_inferences":["The assumption that model states evolve independently may break down in practice because the models share the same pretrained encoder initialization, but the empirical success suggests the method is robust to violations of this assumption.","The asymmetric dual projection could be combined with larger vision-language models (e.g., ViT-L) to further improve absolute retrieval numbers, though this is not explored in the paper.","The SOCM objective is computationally cheaper than centered covariance and provides comparable or slightly better performance; it may be a general improvement for any second-order matching in distillation."],"forward_implications":["Synthetic training data for vision-language models can be reduced by at least 5× without loss of retrieval accuracy at practical budgets.","The staggered model queue paradigm may be applicable to other feature-matching distillation tasks beyond vision-language, such as unimodal dataset distillation or graph condensation.","Because MDA-DD distills data that transfers to unseen visual encoder architectures (ResNet, RegNet, ViT), it may serve as a generic compressed representation for diverse downstream models.","The use of multiple online models eliminates the need for pre-computed expert trajectories, reducing storage and pre-training overhead substantially."],"supporting_citations":[{"why":"Provides the existing state-of-the-art distribution-matching baseline that MDA-DD improves upon.","marker":"[22]"},{"why":"LoRS is a trajectory-matching baseline whose low-rank similarity matrix approach is compared.","marker":"[46]"},{"why":"RepBlend is a trajectory-matching baseline that freezes encoders and blends representations; MDA-DD outperforms it.","marker":"[54]"},{"why":"PTM-ST is a recent staged distillation framework that serves as a strong baseline.","marker":"[16]"}],"fun_headline_variants":["100 synthetic pairs beat 500 on retrieval","Model-distribution-aware distillation tops all prior work","Staggered model queue boosts synthetic data quality","8-model ensemble distills 100 pairs that rival 500"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The theoretical analysis in Appendix C assumes that model states in the dynamic queue evolve independently and that the expected matching loss varies smoothly across training ages (Lipschitz continuity), assumptions not verified in practice and potentially violated due to shared initialization and joint optimization.","fun_headline_variants_meta":{"raw":{"variants":["100 synthetic pairs beat 500 on retrieval","Model-distribution-aware distillation tops all prior work","Staggered model queue boosts synthetic data quality","8-model ensemble distills 100 pairs that rival 500"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"cost":0.000185948,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000185948,"upstream_inference_prompt_cost":8.8788e-05,"upstream_inference_completions_cost":9.716e-05},"total_tokens":1186,"prompt_tokens":839,"completion_tokens":347,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":285}},"tokens_in":839,"tokens_out":347,"cost_source":"openrouter_usage","duration_ms":3570,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":285,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T15:10:08.494298+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a new multimodal distillation method using 100 synthetic pairs fails to exceed the best 500-pair baseline on Flickr30K image-to-text recall@1, then the central claim of 5× data efficiency would be refuted.","supporting_citations":[{"cited_title":"Covmatch: Cross-covariance guided multimodal dataset distillation with trainable text encoder","cited_arxiv_id":null,"evidence_quote":"Provides the existing state-of-the-art distribution-matching baseline that MDA-DD improves upon."},{"cited_title":"Low-rank similarity mining for multimodal dataset distillation","cited_arxiv_id":null,"evidence_quote":"LoRS is a trajectory-matching baseline whose low-rank similarity matrix approach is compared."},{"cited_title":"Beyond modality collapse: Representation blending for multimodal dataset distillation","cited_arxiv_id":null,"evidence_quote":"RepBlend is a trajectory-matching baseline that freezes encoders and blends representations; MDA-DD outperforms it."},{"cited_title":"Multimodal dataset distillation via phased teacher models","cited_arxiv_id":null,"evidence_quote":"PTM-ST is a recent staged distillation framework that serves as a strong baseline."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}