Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that multimodal dataset distillation fails mainly because contrastive supervision, combined with distillation's over-compression, collapses each modality's representations, and that representation blending plus symmetric…

desk verdict Strong empirical MDD paper with a plausible but under-supported theory of modality collapse; the method and results deserve serious review, the theory needs tightening. read the letter →

arxiv 2505.14705 v1 pith:EEKS7JJP submitted 2025-05-16 cs.CV cs.LG

Xin Zhang , Ziruo Zhang , Jiawei Du , Zuozhu Liu , Joey Tianyi Zhou This is my paper · ORCID
classification cs.CVcs.LG
keywords multimodaldatasetdistillationmodalitycollapsecontrastivelearningrepresentationblendingprojectiontrajectorymatchingcross-modalretrievalcondensationvision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the main failure of multimodal dataset distillation is not poor data selection but modality collapse: the distilled image and text embeddings clump tightly within each modality and drift far apart across modalities. It argues this collapse comes from a conflict between dataset distillation, which over-compresses toward a few dominant features, and contrastive cross-modal supervision, which reinforces that concentration. The proposed RepBlend framework weakens the over-dominant cross-modal signal by blending representations within each modality and balances the two branches by matching both projection heads' training trajectories. A sympathetic reader would care because, if right, tiny distilled image-text sets roughly 0.3% of the original data can support cross-modal retrieval much closer to full-data quality, with a substantial speedup in distillation cost.

What carries the argument

The load-bearing identity is Equation 3, which says the inner product of the gradients for two similar image representations is proportional to $w_{nm}w_{mn}[\sigma(\hat y_{nm})/\gamma - \tilde y_{nm}][\sigma(\hat y_{mn})/\gamma - \tilde y_{mn}]\tilde\tau_m^{\prime\top}\tilde\tau_n^{\prime}$, derived under the assumption that negative pairs are mutually orthogonal in high-dimensional embedding spaces; this identity converts 'contrastive loss' into 'intra-modal attraction.' Representation Blending then interleaves shuffled same-modality embeddings through a Beta-distributed interpolation before the contrastive loss is computed, weakening the overly strong cross-modal term. Symmetric Projection Trajectory Matching replaces the asymmetric setup, in which the image encoder was matched against a text projection head, with matched trajectories of both projection heads so that image and text receive comparable supervision.

What would settle it

Run the same distillation in an embedding space of dimension below about 50, where negative pairs are clearly non-orthogonal, and track whether the gradient-alignment term in Equation 3 still predicts the measured growth in intra-modal similarity; if the term stops predicting collapse while blending still helps, the stated cause of modality collapse is wrong even though the remedy may work.

Watch

Extended reading notes

Core claim

Modality collapse is the central object: as distillation progresses, intra-modal cosine similarity among synthetic embeddings rises (from 0.512 to 0.522 in the paper's tracking), and because embeddings are high-dimensional this tiny rise corresponds to a more than twofold increase in the concentration ratio; non-matching cross-modal pairs show near-uniform similarity, so soft labels lose their fine-grained relational information. The paper's derivation shows that the update direction of two similar image representations is dominated by the positive-pair term $\tilde\tau_m^{\prime\top}\tilde\tau_n^{\prime}$ from the weighted binary cross-entropy loss, so contrastive supervision pushes same-modality representations together. To counteract this, RepBlend applies representation blending, a MixUp-style linear interpolation of each representation with a shuffled same-modality instance in the embedding space, and symmetric projection trajectory matching, which adds a trainable image projection head and matches the training trajectories of both image and text projection heads instead of matching a text projection head against a full image encoder. On Flickr-30K and MS-COCO, at 100 synthetic pairs, retrieval at K=10 improves by 9.4 points for image retrieval and 6.3 points for text retrieval over the previous state of the art, with up to 6.7x faster distillation iterations.

Load-bearing premise

The key assumption is that in high-dimensional embedding spaces all non-matching examples are mutually orthogonal and the shared negative examples do not affect the relative update direction of two similar image representations; if real image and text embeddings are not that orthogonal, the paper's causal story for modality collapse is not proven.

Editorial extensions

If this is right

  • If RepBlend is right, distilled image-text sets of 100 to 500 pairs can serve as practical surrogates for cross-modal retrieval, reaching IR@10 of 44.5 and TR@10 of 55.5 on Flickr-30K at 100 pairs.
  • The modality-collapse diagnosis transfers to any dataset-distillation method that optimizes contrastive objectives, not just the specific baseline analyzed, so blending is a candidate drop-in remedy for those methods.
  • Frozen image and text encoders with projection-head trajectory matching cut distillation cost by up to 6.7x per iteration and roughly halve peak GPU memory without sacrificing accuracy.
  • The same mechanism extends beyond image-text pairs, since the paper reports consistent gains on audio-text retrieval when the image encoder is swapped for an audio encoder.
  • RepBlend also produces distilled data usable for classification: on ImageNet-100 at one image per class, the paper reports 65.8% top-1 accuracy versus 2.5% for the comparison method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the orthogonality assumption used in the derivation is strong in real CLIP-style embedding spaces; a direct way to test the mechanism is to repeat the gradient-tracking experiment on low-dimensional embeddings where negative pairs are clearly non-orthogonal, and if collapse still occurs without the predicted alignment term, the explanation needs revision even though RepBlend m
  • Editorial inference: representation blending is essentially MixUp applied in representation space, so a natural cross-check is whether the interpolation strength sampled from Beta(alpha, alpha) should be annealed over distillation iterations rather than fixed per batch.
  • Editorial inference: symmetric projection trajectory matching could transfer to unimodal trajectory-matching distillers that suffer from parameter-count imbalance between the modules being matched.
  • Editorial inference: the paper's concentration-ratio metric, which turns a tiny cosine-similarity rise into a twofold change, suggests that collapse detectors for future distillation work should report cap-area or concentration statistics rather than raw cosine similarity alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper addresses multimodal dataset distillation (MDD), identifying a phenomenon called modality collapse, characterized by over-concentrated intra-modal representations and an enlarged inter-modal gap. The authors propose RepBlend, which combines representation blending (a MixUp-style interpolation in representation space) to weaken cross-modal supervision and symmetric projection trajectory matching to balance optimization dynamics across modalities. Experiments on Flickr-30K and MS-COCO, along with an audio-text extension on AudioCaps, report consistent and large gains over prior MDD baselines, including improved retrieval metrics and substantial computational speedups.

Significance. If the empirical results hold, RepBlend is a practically strong contribution: it achieves state-of-the-art retrieval performance across multiple distillation budgets and architectures, with reported gains such as +9.4 IR@10 and +6.3 TR@10 over LoRS at the 100-pair setting on Flickr-30K, and a 6.7x distillation speedup. The evaluation is broad, covering two standard image-text benchmarks, an audio-text benchmark, multiple budgets, cross-architecture generalization, and hyperparameters inherited from LoRS without fitting to the target retrieval numbers. However, the paper's central causal claim that modality collapse arises from a fundamental conflict between dataset distillation and contrastive supervision is not established by the theoretical derivation, which relies on unproven assumptions. The empirical method and its advantages stand independently, but the theoretical framing needs substantial revision.

major comments (3)
  1. [Section 3.1 and Appendix B, Eq. (3)] The derivation of Eq. (3) rests on two unstated assumptions: (i) all non-self, non-mutual text embeddings are mutually orthogonal in the high-dimensional space, and (ii) the aggregated influence of shared negative examples on the relative update direction is negligible relative to the mutual interaction term. Neither assumption is stated as a formal condition nor tested empirically. In the actual distilled setting |S| is only 100-500, the embeddings are learned rather than random, and LoRS's soft labels are non-uniform, so these conditions may fail. Consequently, the Proposition's conclusion that cross-modal supervision reinforces intra-modal similarity is not established, and the 'for the first time' causal identification in the abstract and Section 1 is not supported by the theory alone. I recommend either stating explicit sufficient conditions and validating them, or reframing the contribution as an empirical observation with a heuristic explanation.
  2. [Section 4.3, Figure 5] The ablation study for Representation Blending and Symmetric Projection Trajectory Matching is referenced as 'Figure 5,' but the figure does not appear in the manuscript provided. Since the individual and combined contributions of the two proposed components are central empirical claims, the figure (or an equivalent table of ablation results) must be included in the final version.
  3. [Section 3.2, Eq. (5)] The representation blending operation is a MixUp-style interpolation in representation space. The paper interprets the gains in Figure 3 as evidence that weakening cross-modal supervision specifically mitigates collapse, but the same operation could also act as a generic regularizer that improves diversity without specifically reducing cross-modal supervision. A control experiment comparing RepBlend against standard representation MixUp under the same wBCE objective would be needed to support the causal interpretation. Without it, the mechanism remains underdetermined.
minor comments (6)
  1. [Abstract and Section 1] The phrase 'at the first time' should be corrected to 'for the first time'.
  2. [Eq. (3) and Appendix B] The notation is inconsistent: Eq. (3) uses a lowercase 't' in the denominator of the sigmoid argument, while the temperature is defined as gamma in Eq. (2) and used throughout Appendix B. Please unify the notation.
  3. [Figure 2 caption] The caption mentions 'concentration ratio (CR)2' with an ambiguous superscript; please clarify whether CR is squared or the '2' is a footnote marker.
  4. [Algorithm 1, line 8] There is a typo: 'representaion' should be 'representation'.
  5. [Section 3.3] The claim that the parameter-count difference between the image encoder and text projection head explains the asymmetric trajectory loss is not directly substantiated. A direct measure such as per-parameter update norms or a controlled experiment varying parameter counts would strengthen this explanation.
  6. [General] In the manuscript version provided, Figures 1-10 appear only as captions without the actual plots or images. The final submission must embed all figures so that the visual evidence (e.g., Figure 1 distributions, Figure 5 ablations) is available to readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the theoretical mechanism is an explicit approximation, and the empirical contributions are evaluated on external benchmarks with inherited hyperparameters.

full rationale

The paper's central derivation chain is not circular. The claim that cross-modal contrastive supervision reinforces intra-modal similarity is derived explicitly in Section 3.1 and Appendix B from the wBCE gradient structure, with the assumptions (high-dimensional mutual orthogonality of negative pairs and negligible aggregated shared-negative influence) stated in the derivation itself. Whether these assumptions hold in the distilled setting is a correctness or robustness concern, not a circularity: the conclusion is not assumed as an input but obtained by an analytic approximation. The proposed remedy (representation blending and symmetric projection trajectory matching) is then validated on Flickr-30K, MS-COCO, and AudioCaps against external baselines, with hyperparameters inherited from LoRS as stated in Section 4.1 and Appendix F; the blending parameter beta_distribution alpha is fixed at 1.0, and no parameter is fitted to the reported retrieval numbers. The perturbation experiment (Section 3.2, Figure 3) is an empirical motivation, not a renamed prediction. Self-citations to the authors' prior dataset-distillation works (e.g., refs. 13-15) support the background premise that DD tends toward dominant features, but that premise is also cited to external works (9, 18, 42) and is not equivalent to the paper's target result of modality collapse in multimodal distillation. No uniqueness theorem is imported from prior work, and no ansatz is smuggled in by citation. The method's empirical gains are therefore self-contained evidence, and the theoretical explanation, while approximate, is independent of the experimental outcome.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The proposed method does not introduce new physical or conceptual entities. It relies on three assumptions: orthogonality of negative pairs, dominance of the mutual-interaction term, and sufficiency of frozen encoder features. The free parameters are conventional hyperparameters, with alpha being the only new one introduced by RepBlend.

free parameters (2)
  • alpha (Beta distribution parameter for representation blending) = 1.0
    Controls the mixing strength in RepBlend (Algorithm 2). Chosen as a hyperparameter, not fitted to the target results; listed in Appendix F, Table 6 as beta_distribution alpha=1.0.
  • beta (similarity threshold in wBCE loss) = not specified in paper, inherited from LoRS
    Threshold that separates positive and negative pairs in Equation 2. Its value is not stated in the paper but inherited from the LoRS baseline; it affects the loss but is not tuned for the proposed claim.
assumptions (3)
  • domain assumption In high-dimensional embedding spaces, all negative image-text pairs are mutually orthogonal (i.e., tau'_i^T tau'_j approx 0 for i != j).
    Used in Appendix B to discard many terms in the gradient dot product derivation. This is a common but unverified heuristic; if it fails, the simplified formula in Equation 3 is not accurate.
  • ad hoc to paper The aggregated influence of shared negative examples on the relative update direction of two similar image representations is negligible compared to the mutual positive-pair term.
    Invoked in Appendix B to drop a large summation term. The paper provides no bound or evidence; this is the key unproven step that makes the Proposition lead to its stated conclusion.
  • domain assumption Frozen pretrained encoders (ImageNet for images, BERT for text) provide representations sufficient for cross-modal alignment during distillation, so only small projection heads need to be trained.
    Central to the design of symmetric projection trajectory matching (Section 3.3 and Algorithm 1). If frozen features are insufficient, the efficiency gain would come at the cost of distillation quality; experiments support it, but it is a prior assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation." pith.science (2026). https://pith.science/paper/EEKS7JJP

@misc{pith2026250514705,
  author       = {Pith},
  title        = {Pith review of: Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EEKS7JJP}},
  note         = {Machine review of arXiv:2505.14705}
}
abstract

Multimodal Dataset Distillation (MDD) seeks to condense large-scale image-text datasets into compact surrogates while retaining their effectiveness for cross-modal learning. Despite recent progress, existing MDD approaches often suffer from \textit{\textbf{Modality Collapse}}, characterized by over-concentrated intra-modal representations and enlarged distributional gap across modalities. In this paper, at the first time, we identify this issue as stemming from a fundamental conflict between the over-compression behavior inherent in dataset distillation and the cross-modal supervision imposed by contrastive objectives. To alleviate modality collapse, we introduce \textbf{RepBlend}, a novel MDD framework that weakens overdominant cross-modal supervision via representation blending, thereby significantly enhancing intra-modal diversity. Additionally, we observe that current MDD methods impose asymmetric supervision across modalities, resulting in biased optimization. To address this, we propose symmetric projection trajectory matching, which synchronizes the optimization dynamics using modality-specific projection heads, thereby promoting balanced supervision and enhancing cross-modal alignment. Experiments on Flickr-30K and MS-COCO show that RepBlend consistently outperforms prior state-of-the-art MDD methods, achieving significant gains in retrieval performance (e.g., +9.4 IR@10, +6.3 TR@10 under the 100-pair setting) and offering up to 6.7$\times$ distillation speedup.

Figures

Figures reproduced from arXiv: 2505.14705 by the authors.

Figure 1
Figure 1. Multimodal embedding distributions across various distillation methods. We extract image [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Increasing intra-modal similarity as distillation progresses. We run optimization for 3000 iterations and track the intra-modal cosine similarity, which increases from 0.512 to 0.522 (red curve). Though small in magnitude, this rise leads to a more than twofold increase in concentration ratio (CR) 2 due to the high dimensionality of the embedding space. Right: Modality collapse undermines the effectiveness of … view at source ↗
Figure 3
Figure 3. As the noise level λ increases, intra￾modal similarity (blue bars) shows a slight de￾cline, while the modality gap (yellow bars) rises markedly. In contrast, our representation blend￾ing (RB) leverages in-distribution samples to si￾multaneously reduce intra-modal similarity and inter-modal gap, effectively mitigating modality collapse during distillation. which indicates that the optimization is guided by positive p… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Current MDD methods adopt asymmetric distillation. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Ablation study of Representation Blending (RB) and Symmetric Projection Trajectory [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance on Flickr-30K with different combinations of image and text encoders. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Performance on MS-COCO with different combinations of image and text encoders. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Synthetic data visualization on ImageNet-100 from our approach and SRe2L when IPC=1. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Flickr-30K before and after distillation. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: MS-COCO before and after distillation. (Left) The original image-text pairs before the distillation. (Right) The image-text pairs after distillation. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions

    cs.CV 2025-02 conditional novelty 3.0 of 10

    A 2023-2025 survey of dataset distillation that organizes matching, generative, decoupling, and selective methods and tabulates ImageNet-scale accuracy comparisons.

Reference graph

Works this paper leans on

65 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Exploring the limits of large scale pre-training

    Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. InInternational Conference on Learning Representations, 2022

    work page 2022

  2. [2]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  3. [3]

    High-performance large-scale image recognition without normalization

    Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. InInternational Conference on Machine Learning, pages 1059–1071. PMLR, 2021

    work page 2021

  4. [4]

    Dataset distillation by matching training trajectories

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. InCVPR, 2022

    work page 2022

  5. [5]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

  6. [6]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  7. [7]

    Scaling up dataset distillation to imagenet- 1k with constant memory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. InICML, 2023

    work page 2023

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, 2009

    2009

Show all 65 references
  1. [9]

    Exploiting inter-sample and inter-feature relations in dataset distillation

    Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. InCVPR, 2024

    2024

  2. [10]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  3. [11]

    Condensing action segmentation datasets via generative network inversion

    Guodong Ding, Rongyu Chen, and Angela Yao. Condensing action segmentation datasets via generative network inversion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    2025

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021

    2021

  5. [13]

    Jiawei Du, Yidi Jiang, Vincent Y . F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. InCVPR, 2023

    2023

  6. [14]

    Sequential subset matching for dataset distillation

    Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. InNeurIPS, 2023

    2023

  7. [15]

    Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment

    Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. InAdv. Neural Inf. Process. Syst. (NeurIPS), 2024

    2024

  8. [16]

    Springer Science & Business Media, 2009

    Reza Zanjirani Farahani and Masoud Hekmatfar.Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009

    2009

  9. [17]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In2017 IEEE international conference on acoustics, speech and signal processing (I...

    2017

  10. [18]

    Towards lossless dataset distillation via difficulty-aligned trajectory matching

    Ziyao Guo, Kai Wang, George Cazenavette, HUI LI, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. InICLR, 2024

    2024

  11. [19]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, 2016

    2016

  12. [20]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  13. [21]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1704.04861, 2017

    Andrew G Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  14. [22]

    Get more for less: Principled data selection for warming up fine-tuning in LLMs

    Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Kumar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in LLMs. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    Audiocaps: Generat- ing captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. InNAACL-HLT, 2019

    2019

  16. [24]

    Dataset condensation with contrastive signals

    Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. InICML, 2022

    2022

  17. [25]

    Selmatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching

    Yongmin Lee and Hye Won Chung. Selmatch: Effectively scaling up dataset distillation via selection-based initialization and partial updates by trajectory matching. InICML, 2024

    2024

  18. [26]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information P...

    2022

  19. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings,...

    2014

  20. [28]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  21. [29]

    Graph distillation with eigenbasis matching

    Yang Liu, Deyu Bo, and Chuan Shi. Graph distillation with eigenbasis matching. InForty-first International Conference on Machine Learning, 2024

    2024

  22. [30]

    Unidetox: Universal detox- ification of large language models via dataset distillation

    Huimin LU, Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. Unidetox: Universal detox- ification of large language models via dataset distillation. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  23. [31]

    Doubly stochastic neighbor embedding on spheres

    Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighbor embedding on spheres. Pattern Recognition Letters, 125:581–587, 2019

    2019

  24. [32]

    Dilm: Distilling dataset into language model for text-level dataset distillation.Journal of Natural Language Processing, 32(1):252–282, 2025

    Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, and Manabu Okumura. Dilm: Distilling dataset into language model for text-level dataset distillation.Journal of Natural Language Processing, 32(1):252–282, 2025

    2025

  25. [33]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  26. [34]

    Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  27. [35]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE international conference on computer vision,...

    2015

  28. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceed- ings of...

    2021

  29. [37]

    Design- ing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020

    2020

  30. [38]

    Liu, Yuri A

    Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstanti- nos N. Plataniotis. DataDAM: Efficient dataset distillation with attention matching. InICCV, 2023

    2023

  31. [39]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  32. [40]

    Efficient large-scale audio tagging via transformer-to-cnn knowledge distillation

    Florian Schmid, Khaled Koutini, and Gerhard Widmer. Efficient large-scale audio tagging via transformer-to-cnn knowledge distillation. InICASSP 2023-2023 IEEE international Conference on acoustics, Speech and signal processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  33. [41]

    Generalized large-scale data condensation via various backbone and statistical matching

    Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. InCVPR, 2024

    2024

  34. [42]

    Delt: A simple diversity-driven earlylate training for dataset distillation

    Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, and Shitong Shao. Delt: A simple diversity-driven earlylate training for dataset distillation. InCVPR, 2025

    2025

  35. [43]

    Loss- curvature matching for dataset selection and condensation

    Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Loss- curvature matching for dataset selection and condensation. InAISTAS, 2023

    2023

  36. [44]

    On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. InCVPR, 2024

    2024

  37. [45]

    An empirical study of example forgetting during deep neural network learning.arXiv preprint arXiv:1812.05159, 2018

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning.arXiv preprint arXiv:1812.05159, 2018

    2018 arXiv

  38. [46]

    Cafe: Learning to condense dataset by aligning features

    Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. InCVPR, 2022

    2022

  39. [47]

    Dataset distillation with neural characteristic function: A minmax perspective

    Shaobo Wang, Yicun Yang, Zhiyuan Liu, Chenghao Sun, Xuming Hu, Conghui He, and Linfeng Zhang. Dataset distillation with neural characteristic function: A minmax perspective. InCVPR, 2025

    2025

  40. [48]

    Dataset distillation

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018

    2018 arXiv

  41. [49]

    Dancing with still images: video distillation via static-dynamic disentanglement

    Ziyu Wang, Yue Xu, Cewu Lu, and Yong-Lu Li. Dancing with still images: video distillation via static-dynamic disentanglement. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6296–6304, 2024

    2024

  42. [50]

    Herding dynamical weights to learn

    Max Welling. Herding dynamical weights to learn. InProceedings of the 26th Annual Interna- tional Conference on Machine Learning, pages 1121–1128, 2009

    2009

  43. [51]

    Vision-language dataset distillation

    Xindi Wu, Byron Zhang, Zhiwei Deng, and Olga Russakovsky. Vision-language dataset distillation. InTMLR, 2024

    2024

  44. [52]

    Low-rank similarity mining for multimodal dataset distillation

    Yue Xu, Zhilin Lin, Yusong Qiu, Cewu Lu, and Yong-Lu Li. Low-rank similarity mining for multimodal dataset distillation. InICML, 2024

    2024

  45. [53]

    Dataset distillation in large data era

    Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. 2023

    2023

  46. [54]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. InNeurIPS, 2024. 13

    2024

  47. [55]

    Navigating complexity: Toward lossless graph condensation via expanding window matching

    Yuchen Zhang, Tianle Zhang, Kai Wang, Ziyao Guo, Yuxuan Liang, Xavier Bresson, Wei Jin, and Yang You. Navigating complexity: Toward lossless graph condensation via expanding window matching. InForty-first International Conference on Machine Learning, 2024

    2024

  48. [56]

    Dataset condensation with differentiable siamese augmentation

    Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, 2021

    2021

  49. [57]

    Dataset condensation with distribution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. InWACV, 2023

    2023

  50. [58]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. InICLR, 2021. 14 A More Related Works Dataset distillation (DD), first proposed by Wang et al. [48], aims to improve training efficiency by condensing information from large-scale datase...

    2021

  51. [59]

    While this approach is unbiased, it may fail to capture the most informative or representative instances necessary for efficient training

    Random (Rand): Randomly selects a subset of samples from the full dataset to form a coreset. While this approach is unbiased, it may fail to capture the most informative or representative instances necessary for efficient training

  52. [60]

    It iteratively chooses instances that minimize the discrepancy between the coreset and the full dataset’s feature distribution

    Herding (Herd) [50]: Selects samples based on herding dynamics to approximate the mean of the data distribution. It iteratively chooses instances that minimize the discrepancy between the coreset and the full dataset’s feature distribution. 17 Table 5: Hyperparameter settings ...

    2000

  53. [61]

    It aims to maximize coverage by iteratively choosing points that are maximally distant from the already selected ones

    K-Center (K-Cent) [16]: Selects samples that serve as representative centers in the feature space. It aims to maximize coverage by iteratively choosing points that are maximally distant from the already selected ones

  54. [62]

    Samples with low forgetting counts are removed first, prioritizing the retention of harder and more informative examples

    Forgetting (Forget) [45]: Selects samples based on how often they are forgotten during training, i.e., when correct predictions become incorrect. Samples with low forgetting counts are removed first, prioritizing the retention of harder and more informative examples. Dataset D...

  55. [63]

    MTT-VL [51]: The first MDD approach that extends the trajectory matching framework MTT [4] to vision-language data, enabling dataset distillation in multimodal settings

  56. [64]

    TESLA-VL [ 52]: An efficient variant of the MTT framework, TESLA [ 7], implemented in LoRS [52] as an ablation to evaluate the effectiveness of similarity mining in multimodal distillation

  57. [65]

    A picture of [ClassName]

    LoRS [52]: A sota MDD method that distills both image-text pairs and their similarity matrix to enhance multimodal distillation, while leveraging low-rank factorization for improving efficiency. F Hyperparameter Settings The hyperparameter settings, summarized in Table 5 and T...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.