REVIEW 3 major objections 5 minor 2 cited by
Video Set Distillation: Information Diversification and Temporal Densification
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims IDTD is the first video-set distillation method to cut both within-clip and between-clip redundancy at once, reporting state-of-the-art accuracy on four action datasets.
desk verdict The paper's claimed inter-sample diversification mechanism is not present in its own formalism; what remains is a within-video diversity method with solid gains at higher IPC but overclaimed SOTA. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the division of labor between a shared per-class Feature Pool and K feature-specific Selectors, supervised by two competing losses. Each Selector is a lightweight linear layer that transforms the pool into a Diverse Segment, and the diversity loss $\mathcal{L}_{div}$ maximizes pairwise $L_2$ distances between segment features, so the Selectors spread out while the pool preserves class-common knowledge. The Temporal Fusor is a convolution along time that fuses the K segments into one video $v_{m,n} = f_{m,n}(\{d_{m,n,k}\})$, and Stochastic Temporal Augmentation $\tau(\cdot)$ randomly crops a temporal interval of $v_{m,n}$ and reshapes it to the target length, preventing fixed temporal positions and forcing the synthetic video to be temporally dense. The overall objective combines a dataset-distillation matching loss on the diverse segments, the diversity loss, and a second matching loss on the augmented synthetic video; the paper uses a ConvNet3D student model's features to compute these losses.
What would settle it
Train a video classifier with a different architecture (for example, a video transformer or a different 3D CNN) on the synthetic set produced by IDTD and evaluate on the same test split: if accuracy collapses to near random while the same synthetic set trains ConvNet3D well, the distilled set encodes extractor-specific features rather than general temporal action information. A cheaper check is to remove Stochastic Temporal Augmentation and retrain; a large drop would indicate the augmentation, not the synthetic content, is responsible for the reported accuracy.
Extended reading notes
Core claim
The paper's central claim is that a video set can be distilled by jointly removing within-sample and inter-sample redundancy, and that the two must be optimized together rather than sequentially. On the author's terms, IDTD builds each synthetic video from K diverse segments: K randomly initialized linear Feature Selectors read a shared learnable Feature Pool and emit segments $d_{m,n,k} = s_{m,n,k}(p_{m,n})$; a diversity loss pushes the segment features apart in $L_2$ distance while a dataset-distillation matching loss aligns them with real data. A Temporal Fusor, a light convolution over time, stitches the segments into a single synthetic video of the target temporal size, and Stochastic Temporal Augmentation randomly samples a temporal interval and rescales it to that size during training. The paper reports that this two-part mechanism outperforms image-level distillation baselines and the static-image-based VDSD on all four evaluated datasets.
Load-bearing premise
The method assumes that a shared per-class Feature Pool plus K random linear Feature Selectors and a diversity loss can represent the information distribution of a real video class, and that the matching and diversity losses computed with ConvNet3D features guarantee that synthetic videos will train a same-architecture student to generalize to real test data; no cross-architecture transfer is reported.
Editorial extensions
If this is right
- IDTD's synthetic videos can replace a large real video set for training recognition models, cutting storage and compute while staying competitive with or better than prior distillation outputs.
- Larger Instance Per Class budgets yield larger accuracy gains, so the method becomes more attractive as the allowed synthetic set size grows.
- Classes with high within-sample and inter-sample redundancy benefit most, so the method is best suited to repetitive, temporally redundant video collections.
- The joint objective is compatible with existing dataset distillation matching losses, so IDTD can be layered onto different distillation objectives rather than replacing them.
Reading between the lines
- An editorial inference: the Feature Pool / Feature Selector decomposition could generalize to other nested-set data (multi-view images, point-cloud sequences) where within-sample and inter-sample redundancy co-occur; a testable extension would apply the same two-loss structure there.
- An editorial inference: because evaluation is only reported on ConvNet3D, an open question is whether the synthetic videos carry class information to other architectures; a cross-architecture benchmark would settle whether the features are genuinely video-level or extractor-specific.
- An editorial inference: Stochastic Temporal Augmentation resembles a temporal dropout regularizer; ablating it while fixing the rest would reveal how much of the accuracy gain comes from augmentation rather than from the synthesized content.
- An editorial inference: the diversity loss operates in feature space only; adding a pixel- or trajectory-space diversity penalty might further improve transferability, but the paper does not explore this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Video Set Distillation as a new problem and proposes IDTD (Information Diversification and Temporal Densification), a method that synthesizes a compact set of videos from a real video dataset. The method uses per-instance Feature Pools and Feature Selectors to generate K diverse segments, a Temporal Fusor to combine them into a synthetic video, and a Stochastic Temporal Augmentation during training. The overall objective combines a dataset distillation matching loss on the segments and on the fused video with a diversity loss on the segments. The authors claim that IDTD jointly reduces within-sample and inter-sample redundancy and report state-of-the-art results on MiniUCF, HMDB51, SSv2, and K400, with ablations and qualitative results.
Significance. If the central claim is supported, this would be a valuable new direction for dataset distillation: it explicitly targets the two-layer nested redundancy of video sets, which prior image-level distillation methods and VDSD do not jointly address. The paper provides a modular design, experiments on four datasets, and ablations that isolate the contributions of the modules and losses. The empirical gains at higher IPC on the small-scale datasets are encouraging. However, the formal description of the method raises a load-bearing concern: as written, the mechanism does not actually minimize inter-sample redundancy, and the reported tables contradict the unqualified state-of-the-art claim in the abstract. These issues need to be resolved before the paper's main contribution can be accepted.
major comments (3)
- [Section 3.2 and Algorithm 1] The formal definition P = {{p_{m,n}}_M}_N and S = {{{s_{m,n,k}}_K}_M}_N, together with the update rule d_{m,n,k} = s_{m,n,k}(p_{m,n}) in Algorithm 1, describes a separate Feature Pool p_{m,n} for each synthetic instance m of class n, not a per-class shared pool. The diversity loss in Eq. (1) compares only segments within the same instance (same m and n), so no term in the objective promotes diversity across different synthetic instances m and m' or shares information across them. Consequently, the claimed joint minimization of inter-sample redundancy is not realized by the written method; as written, the method reduces within-sample redundancy and then independently optimizes each synthetic video. The authors need to either change the formulation to a per-class shared pool and add an explicit inter-instance diversity mechanism, or substantiate with analysis and experiments how the per-instance formulation reduces inter-sample redundancy. This issue is load-bearing for the paper's central claim.
- [Section 3.4, Eq. (2)] The training objective L = L_M((D, T) + alpha1 * L_div + alpha2 * L_M((V'_syn, T) has unbalanced parentheses and cannot be parsed as written. In addition, L_M is never concretely defined; the text only says it is a 'dataset distillation matching objective function between synthetic and real videos'. The sentence immediately after Eq. (2) also says 'within-sample redundancy and inter-class redundancy are jointly reduced', which should be 'inter-sample redundancy'. Please provide a well-formed objective and a precise definition of L_M, since this equation is central to the method and to reproducibility.
- [Table 1 and Table 2] The abstract states that the method 'achieves state-of-the-art results in Video Dataset Distillation', but the reported numbers contradict this at IPC=1. On MiniUCF, MTT+VDSD achieves 23.3±0.6 while IDTD achieves 22.52±0.1; on K400, MTT+VDSD and DM+VDSD achieve 6.3±0.1 and 6.3±0.2 while IDTD achieves 6.1±0.1; on SSv2, MTT+VDSD achieves 5.5±0.1 while IDTD achieves 3.9±0.1. The SOTA claim must be qualified to the settings where it actually holds (higher IPC), or the text must explain why these IPC=1 cases are excluded from the claim.
minor comments (5)
- [Eq. (1)] The upper limit of the outer sum is K-2, which for K segments omits the pair (K-1, K); the limit should presumably be K-1. The notation also does not show m and n indices on L_div, which makes it less clear that the loss is applied per instance.
- [Algorithm 1] Line 16 says 'Calculate loss by equation 1', but Eq. (1) is only the diversity loss; the update should use the total loss from Eq. (2). Also, Algorithm 1 calls tau(v_{m,n}) while Section 3.3 defines tau(v_{m,n}, mu); the two formulations should be consistent.
- [Figure 3] The caption cites 'SOTA [37]', but the reference list ends at [26]; the citation either needs a corresponding reference or should be removed.
- [Section 4.1] There are several typos, including 'Kenetics400' (should be Kinetics-400), 'Diversificaiton' in the Figure 2 caption, and 'optimzition' in Section 4.5.
- [Section 4.2 and evaluation] No cross-architecture transfer experiment is reported: the student model in the matching loss and the evaluated model are both ConvNet3D. An experiment that trains a different video architecture on the same synthetic set would strengthen the claim that the synthetic videos are generally useful, but its absence is not fatal for the same-architecture comparison.
Circularity Check
No significant circularity: the accuracy claims are evaluated on held-out real test data, and no load-bearing self-citation chain is present.
full rationale
The derivation chain in this paper does not reduce to its own inputs. The synthetic videos are optimized using a diversity loss Ldiv and dataset-distillation matching losses LM, but the reported accuracies are obtained by training a student model on the synthetic videos and evaluating on the original held-out test sets (Section 4.3). That evaluation is external and not defined in terms of the training losses, so the performance claims are falsifiable rather than tautological. The diversity loss does enforce segment diversity by construction, but this is a training objective, not a predicted result; the paper does not claim downstream accuracy follows solely from Ldiv. No load-bearing self-citations appear: the baselines (VDSD, DM, MTT, FRePo) are prior external works, and no uniqueness theorem or prior result by the same authors is invoked to force the method. The notation in Section 3.2 and Algorithm 1 defines a separate Feature Pool per synthetic instance (p_{m,n}), so the text's claim of a per-class shared pool may be internally inconsistent and the inter-sample diversification mechanism may be weaker than claimed; however, this is a correctness or consistency risk, not a circularity, because the final evaluation remains external. The ablation studies compare against internal variants and external baselines, providing additional non-circular evidence for the contribution. Overall, the central claim is independently testable and no specific step reduces by construction or by self-citation.
Assumptions & free parameters
free parameters (5)
- alpha1 (diversity loss weight) =
0.05
- alpha2 (temporal fusor matching loss weight) =
1e-4
- K (number of Feature Selectors / Diverse Segments per synthetic video) =
8
- learning rate =
0.01
- synthetic video temporal size =
not explicitly stated; follows VDSD and is ablated in Fig.3
assumptions (3)
- domain assumption Feature statistics of a student model (ConvNet3D) are sufficient statistics for aligning synthetic and real video distributions.
- ad hoc to paper A shared per-class Feature Pool with K randomly initialized linear Feature Selectors can represent the intra-class diversity of a video dataset.
- domain assumption Stochastic Temporal Augmentation, which randomly samples a temporal interval and resizes it, preserves or improves the information content of synthetic videos.
Cite this review
Pith. "Pith review of Video Set Distillation: Information Diversification and Temporal Densification." pith.science (2026). https://pith.science/paper/RCWSKNQ5
@misc{pith2026241200111,
author = {Pith},
title = {Pith review of: Video Set Distillation: Information Diversification and Temporal Densification},
year = {2026},
howpublished = {\url{https://pith.science/paper/RCWSKNQ5}},
note = {Machine review of arXiv:2412.00111}
}
read the original abstract
The rapid development of AI models has led to a growing emphasis on enhancing their capabilities for complex input data such as videos. While large-scale video datasets have been introduced to support this growth, the unique challenges of reducing redundancies in video \textbf{sets} have not been explored. Compared to image datasets or individual videos, video \textbf{sets} have a two-layer nested structure, where the outer layer is the collection of individual videos, and the inner layer contains the correlations among frame-level data points to provide temporal information. Video \textbf{sets} have two dimensions of redundancies: within-sample and inter-sample redundancies. Existing methods like key frame selection, dataset pruning or dataset distillation are not addressing the unique challenge of video sets since they aimed at reducing redundancies in only one of the dimensions. In this work, we are the first to study Video Set Distillation, which synthesizes optimized video data by jointly addressing within-sample and inter-sample redundancies. Our Information Diversification and Temporal Densification (IDTD) method jointly reduces redundancies across both dimensions. This is achieved through a Feature Pool and Feature Selectors mechanism to preserve inter-sample diversity, alongside a Temporal Fusor that maintains temporal information density within synthesized videos. Our method achieves state-of-the-art results in Video Dataset Distillation, paving the way for more effective redundancy reduction and efficient AI model training on video datasets.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 2 Pith papers
-
Temporal Saliency-Guided Distillation: A Scalable Framework for Distilling Video Datasets
Temporal saliency masks computed from inter-frame differences guide gradient updates and augmentation in a uni-level video dataset distillation framework, achieving state-of-the-art results on MiniUCF, HMDB51, Kinetic...
-
Latent Video Dataset Distillation
Latent-space encoding with DPP selection and HOSVD compression outperforms prior video dataset distillation methods on MiniUCF, HMDB51, Kinetics-400, and SSv2 at IPC 1 and 5.
Reference graph
Works this paper leans on
-
[1]
Quo vadis and action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman. Quo vadis and action recognition? a new model and the kinetics dataset. In CVPR,
-
[2]
Dataset distillation by matching training trajectories
George Cazenavette, Tongzhou Wang, Alexei A Efros An- tonio Torralba, , and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022. 2, 6
work page 2022
-
[3]
Scal- ing up dataset distillation to imagenet-1k with constant mem- ory
Justin Cui, Ruochen Wang, Si Si, , and Cho-Jui Hsieh. Scal- ing up dataset distillation to imagenet-1k with constant mem- ory. In IMCL, 2023. 3
work page 2023
-
[4]
F. Dirfaux. Key frame selection to represent a video. In Proceedings 2000 International Conference on Image Pro- cessing, 2000. 2
work page 2000
-
[5]
Spatiotemporal residual networks for video action recogni- tion
Christoph Feichtenhofer, Axel Pinz, , and Richard P Wildes. Spatiotemporal residual networks for video action recogni- tion. In NeurIPS, 2016. 3
work page 2016
-
[6]
Slowfast networks for video recognition
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. 3
work page 2019
-
[7]
The ”something something” video database for learning and evaluating visual common sense
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal ski, Joanna Materzy ´ nska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video database for learning and evaluating visual common sense. In ICCV, 2017. 1, 5
work page 2017
-
[8]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,
Show all 26 references
-
[9]
Large-scale video classification with convolutional neural networks
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR,
-
[10]
Kuehne, H
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, , and T. Serre. Hmdb: A large video database for human motion recogni tion. In ICCV, 2011. 1, 5
2011
-
[11]
Data distillation for text classifica- tion
Yongqi Li and Wenjie Li. Data distillation for text classifica- tion. In arXiv preprint arXiv:2104.08448, 2021. 3
2021 arXiv
-
[12]
Herding dynamical weights to learn
MaxWelling. Herding dynamical weights to learn. In ICML,
-
[13]
Active learning for con volutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese. Active learning for con volutional neural networks: A core-set approach. In ICLR,
-
[14]
Ucf101: A dataset of 101 human actions classes from videos in the wild
Khurram Soomro, Amir Roshan Zamir, , and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. In CoRR, 2012. 1, 5
2012
-
[15]
On the diversity and realism of distilled dataset: Anefficient dataset distilla- tion paradigm
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: Anefficient dataset distilla- tion paradigm. In CVPR, 2024. 3
2024
-
[16]
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015. 3
2015
-
[17]
Cafe: Learning to condense dataset by aligning features
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Wang Shuo Yang, Guan Huang, Hakan Bilen, Xinchao Wang, , and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022. 2
2022
-
[18]
Dataset distillation
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, , and Alexei A Efros. Dataset distillation. In arXiv preprint arXiv:1811.10959, 2018. 2
2018 arXiv
-
[19]
Dancing with still images: Video distillation via static-dynamic dis- entanglement
Ziyu Wang, Yue Xu, Cewu Lu, and Yong-Lu Li. Dancing with still images: Video distillation via static-dynamic dis- entanglement. In CVPR, 2024. 2, 3, 5, 6, 7, 8
2024
-
[20]
Multi- modal dataset distillation for image-text retrieval
Xindi Wu, Zhiwei Deng, , and Olga Russakovsky. Multi- modal dataset distillation for image-text retrieval. In arXiv preprint arXiv:2308.07545, 2023. 3
2023 arXiv
-
[21]
Dataset pruning: Reducing training data by ex- amining generalization influence
Shuo Yang, ZekeXie HanyuPeng, MinXu, MingmingSun, and PingLi. Dataset pruning: Reducing training data by ex- amining generalization influence. In Proceedings 2000 In- ternational Conference on Image Processing, 2000. 2
2000
-
[22]
Squeeze and recover and relabel: Dataset condensation at imagenet scale from a new perspective
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze and recover and relabel: Dataset condensation at imagenet scale from a new perspective. In NeurIPS, 2023. 3
2023
-
[23]
Dataset condensation with distri- bution matching
Bo Zhao and Hakan Bilen. Dataset condensation with distri- bution matching. In WACV, 2023. 2, 6, 8
2023
-
[24]
Dataset condensation with gradient matching
Bo Zhao, Konda Reddy Mopuri, , and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021. 2
2021
-
[25]
Im- proved distribution matching for dataset condensation
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Im- proved distribution matching for dataset condensation. In CVPR, 2023. 2
2023
-
[26]
Dataset distillation using neural feature regression
Yongchao Zhou, Ehsan Nezhadarya, , and Jimmy Ba. Dataset distillation using neural feature regression. In NeurIPS, 2022. 2, 6
2022
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.