REVIEW 4 major objections 6 minor 49 references
Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that video dataset distillation should adapt temporal resolution per action class, and that a Q-learning agent rewarded by teacher-scored feature distance can find these resolutions efficiently.
desk verdict Novel per-class temporal resolution for video distillation, but the unvalidated early-stage reward and the SSv2 IPC=1 counterexample mean the SOTA claim is too strong. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the teacher-in-the-loop reward function combined with the Q-learning policy update. The teacher $\Phi_T$ maps synthetic and real videos of the same class into a feature space, and the inverse-distance reward $R_{DAViD}$ measures whether a candidate temporal resolution lets the distilled video preserve the class's semantic content. Because the distillation uses a Distribution Matching loss, the reward can be computed after only $\beta=2\%$ of the distillation iterations, which makes the RL search tractable; the paper reports this reduces wall-clock GPU time by roughly two orders of magnitude compared with grid search over temporal resolutions.
What would settle it
Measure the correlation between $R_{DAViD}$ computed at $N_{early}=\mathrm{int}(0.02N)$ and the final student test accuracy for each candidate temporal resolution across many classes. If a resolution that scores well early routinely finishes worse than one that scores poorly early, the RL policy is chasing noise and the reported gains would not replicate outside the tested settings.
Extended reading notes
Core claim
DAViD claims to be the first method to learn a temporal resolution per semantic class in video dataset distillation. Its central discovery is that dynamic video classes such as golf swings and discus throws preserve their meaning only at high temporal resolution, while static classes such as push-ups and hair drying can be represented with far fewer frames without losing recognition accuracy. The method encodes this into an RL formulation: the action space is the set of candidate temporal resolutions across classes, the state is the current temporal resolution assignment, and the reward is $R_{DAViD}=1/(1+\|\Phi_T(V_{syn})-\Phi_T(V_{real})\|_2)$, where $\Phi_T$ is a teacher model trained on the full dataset. After a short early-stage distillation run, the reward is computed and the Q-table is updated; once the policy converges, the synthetic videos are distilled at the chosen optimal temporal resolution.
Load-bearing premise
The whole scheme depends on the assumption that the teacher-scored feature distance after only about 2% of the distillation iterations reliably predicts which temporal resolution will produce the best final distilled dataset.
Editorial extensions
If this is right
- Dynamic classes such as golf swing or discus throw are assigned more frames per synthetic video, while static classes such as push-ups or hair drying are assigned fewer, so the storage budget adapts to each class's temporal redundancy.
- The class-wise accuracy profile becomes more balanced, removing the performance collapse on dynamic classes that the paper observes in prior video DD methods.
- Finding the optimal temporal resolutions costs roughly two orders of magnitude less wall-clock GPU time than grid search, because the reward is computed after only the early 2% of distillation iterations.
- The distilled synthetic features lie closer to the real-data feature distribution, including for minority or outlying classes, as shown in the paper's t-SNE visualization.
- If the method is correct, temporal resolution in video dataset distillation should be treated as a learnable, semantics-dependent hyperparameter rather than a fixed global constant.
Reading between the lines
- A natural extension would apply the same teacher-reward RL search to spatial resolution or to the number of synthetic videos per class, since the reward mechanism does not depend on which resolution axis is being varied.
- The early-reward trick suggests a broader protocol: any distillation loss whose early values correlate with final accuracy could drive a cheap RL search, not only Distribution Matching.
- The class-wise dynamic score $\Delta$ used to construct TR-UCF could be reused as a prior to initialize the Q-table, potentially reducing exploration and making the search scale to datasets with hundreds of classes.
- The larger gains at higher IPC on SSv2 and K400 suggest the approach will matter more as video distillation is pushed to larger, more diverse real-world datasets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Dynamic-Aware Video Distillation (DAViD), a Q-learning framework that selects a temporal resolution for each class of synthetic videos in video dataset distillation. The reward is a teacher-model feature distance between synthetic and real videos, computed after a small fraction (β=2%) of the distillation iterations; after policy learning, the synthetic videos are distilled at the selected per-class resolutions using distribution matching. Experiments are reported on TR-UCF (a newly proposed dynamic-class subset of UCF101), HMDB51, UCF101, SSv2, and K400 at IPC=1 and 5. The paper claims state-of-the-art results and an efficiency gain of two orders of magnitude over grid search.
Significance. If the central claim holds, DAViD would be a useful contribution: it identifies a real gap in video distillation, proposes a new benchmark (TR-UCF), and the ablation in Table 4 isolates the contribution of semantics-adaptive temporal resolution. The work is also transparent about memory constraints and uses standard evaluation. However, the central claim rests on the early-stage reward proxy and on the action-space definition, both of which are insufficiently validated in the current manuscript.
major comments (4)
- [Section 3.4, Eq. (5)] The reward R_DAViD is computed after N_early = int(0.02*N), i.e., 100 of 5000 DM iterations, but the paper provides no evidence that this early feature distance predicts the final accuracy of the distilled dataset. Table 3 reports only wall-clock time, not the final student accuracy of policies found with early-stage versus full-convergence rewards. This matters mechanistically: a video partitioned into more temporal segments has more synthetic parameters to fit, so after 100 iterations its feature distance may remain large even though it would be the better choice at convergence; if so, the reward systematically penalizes the high temporal resolutions that dynamic classes need. Please report the learned A_T values and add an experiment comparing policies selected by early reward against policies selected by full-convergence reward.
- [Section 3.3] The action space A is defined only abstractly as A={a1,...,an}; the paper never states the concrete temporal-resolution values, the number of options n, or how A_T maps to the number of frames in P(V_Syn, A_t). This is load-bearing because the entire claim is that the RL selects a meaningful per-class resolution; without this information the ablation in Table 4 and the reported accuracies cannot be reproduced or interpreted.
- [Table 2] For SSv2 at IPC=1, DAViD reports 3.5±0.5, which is below both VDSD (5.5±0.1) and MTT (3.9±0.1); the text in §4.4.2 nonetheless states that 'our approach achieved SOTA performance' and claims stronger effectiveness on the heavy-weight track. This is an internal inconsistency that should be corrected or explained.
- [Section 3.3, Eq. (2)] The Q-learning formulation uses a state space 'identical to the action space' but Eq. (2) includes S_{t+1}; no transition dynamics are defined. It is unclear whether this is a contextual-bandit setting (where S_t is constant) or a sequential MDP. If it is the former, the Q-learning update should be simplified and justified; if the latter, the transition function is missing.
minor comments (6)
- [Section 4.1.1] The dataset name is misspelled as 'Kenetics400'; it should be 'Kinetics400'.
- [Section 4.2 vs Table 3] Section 4.2 reports the optimization was conducted on three RTX 3090 GPUs, while Table 3 reports wall-clock GPU time on a single RTX 3090 GPU; please clarify which setup applies to each experiment.
- [Section 4.5.2, Table 4] Please report the standard deviations for Case A and Case B in Table 4 or state if they are single runs; the absence of variance makes the 15.4 vs 20.0 vs 26.6 comparison harder to assess.
- [Section 3.3, Eq. (4)] The operator 'TemporalResize' appears without a definition of the interpolation or repeat scheme; please specify exactly how segments are resized back to the input length.
- [References] Several reference entries are malformed (e.g., '[5]' has a stray comma, '[38]' is missing volume/page details); please clean the bibliography.
- [Figure 1] The x-axis label 'Performance Degradation' with the numbers '200 40 60 80' appears garbled; please check the figure formatting.
Circularity Check
No significant circularity: the reward is an internal training-time proxy and final claims are evaluated on held-out test accuracy against external baselines.
full rationale
DAViD's derivation chain is empirical rather than formal. The teacher-in-the-loop reward R_DAViD (Eq. 1) is computed from the feature distance between synthetic and real training videos and is used only as a training signal for the Q-learning agent that selects temporal resolutions; it is never fitted to, replaced by, or defined in terms of the test accuracy reported in Tables 1-2. The final student models are evaluated on original held-out test sets, so the claimed SOTA results are not the same quantity as the reward by construction. The early-stage truncation at beta=0.02 (Eq. 5) is a computational approximation whose reliability is a legitimate correctness risk, not circularity: a biased reward would yield a worse policy, but the held-out test evaluation would still be an independent check. The TR-UCF split (Eq. 6) is constructed by selecting classes with high sensitivity to temporal information, which shapes the evaluation benchmark, but the method's advantage is also shown on full UCF101, HMDB51, SSv2, and K400, so the central claim does not reduce to the construction of that benchmark. No load-bearing self-citations or imported uniqueness theorems appear; references to Q-learning and distribution matching are standard external results. Therefore no circular step can be exhibited with the required equation-to-equation or fit-to-prediction reduction.
Assumptions & free parameters
free parameters (6)
- beta (early-stage DD ratio) =
0.02
- alpha (Q-learning learning rate) =
0.1
- gamma (discount factor) =
0.5
- p (exploration probability) =
not reported
- Temporal resolution action space A =
not specified
- Teacher model Phi_T =
architecture and training not specified
assumptions (5)
- domain assumption A teacher trained on the original full dataset provides a feature space in which L2 distance between synthetic and real videos measures distillation quality.
- domain assumption Early-stage DM optimization (N_early = beta*N iterations) yields synthetic videos whose feature distance is a reliable reward for policy learning.
- domain assumption Distribution Matching loss is a sufficient backbone for video DD and its optimization landscape responds to temporal resolution changes.
- domain assumption Uniform temporal cropping followed by interleave-repeat resizing preserves the class semantics of videos sufficiently for distillation.
- standard math Q-learning converges for this environment and the state-action formulation is well-posed.
Cite this review
Pith. "Pith review of Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics." pith.science (2026). https://pith.science/paper/MGZZY4YO
@misc{pith2026250602021,
author = {Pith},
title = {Pith review of: Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics},
year = {2026},
howpublished = {\url{https://pith.science/paper/MGZZY4YO}},
note = {Machine review of arXiv:2506.02021}
}
read the original abstract
With the rapid development of vision tasks and the scaling on datasets and models, redundancy reduction in vision datasets has become a key area of research. To address this issue, dataset distillation (DD) has emerged as a promising approach to generating highly compact synthetic datasets with significantly less redundancy while preserving essential information. However, while DD has been extensively studied for image datasets, DD on video datasets remains underexplored. Video datasets present unique challenges due to the presence of temporal information and varying levels of redundancy across different classes. Existing DD approaches assume a uniform level of temporal redundancy across all different video semantics, which limits their effectiveness on video datasets. In this work, we propose Dynamic-Aware Video Distillation (DAViD), a Reinforcement Learning (RL) approach to predict the optimal Temporal Resolution of the synthetic videos. A teacher-in-the-loop reward function is proposed to update the RL agent policy. To the best of our knowledge, this is the first study to introduce adaptive temporal resolution based on video semantics in video dataset distillation. Our approach significantly outperforms existing DD methods, demonstrating substantial improvements in performance. This work paves the way for future research on more efficient and semantic-adaptive video dataset distillation research.
Figures
Reference graph
Works this paper leans on
-
[1]
Youtube-8m: A large- scale video classification benchmark
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large- scale video classification benchmark. InarXiv preprint arXiv:1609.08675, 2016
arXiv 2016
-
[2]
Is space-time attention all you need for video understanding? InICML, 2021
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? InICML, 2021
2021
-
[3]
Apostolos N. Burnetas and Michael N. Katehakis. Optimal adaptive policies for markov decision processes. InMathe- matics of Operation Research, 1997
work page 1997
-
[4]
Quo vadis and action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman. Quo vadis and action recognition? a new model and the kinetics dataset. InCVPR, 2017
work page 2017
-
[5]
Dataset distillation by matching training trajectories
George Cazenavette, Tongzhou Wang, Alexei A Efros An- tonio Torralba, , and Jun-Yan Zhu. Dataset distillation by matching training trajectories. InCVPR, 2022
work page 2022
-
[6]
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Generalizing dataset dis- tillation via deep generative prior. InCVPR, 2023
work page 2023
-
[7]
Scal- ing up dataset distillation to imagenet-1k with constant mem- ory
Justin Cui, Ruochen Wang, Si Si, , and Cho-Jui Hsieh. Scal- ing up dataset distillation to imagenet-1k with constant mem- ory. InIMCL, 2023
work page 2023
-
[8]
F. Dirfaux. Key frame selection to represent a video. In Proceedings 2000 International Conference on Image Pro- cessing, 2000
work page 2000
Show all 49 references
-
[9]
Slowfast networks for video recognition
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019
2019
-
[10]
The ”something something” video database for learning and evaluating visual common sense
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal ski, Joanna Materzy´nska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video dat...
2017
-
[11]
Ross, Carl V ondrick, Car- oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, and Susanna Ricco
Chunhui Gu, Chen Sun, David A. Ross, Carl V ondrick, Car- oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, and Susanna Ricco. Ava: A video dataset of spatio-temporally localized atomic visual actions. InCVPR, 2018
2018
-
[12]
Efficient dataset distillation via minimax diffusion
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Hao- nan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. InCVPR, 2024
2024
-
[13]
Read, watch, and move: Reinforcement learning for temporally grounding natural language descrip- tions in videos
Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen. Read, watch, and move: Reinforcement learning for temporally grounding natural language descrip- tions in videos. InAAAI, 2019
2019
-
[14]
A novel key-frames se- lection framework for comprehensive video summarization
Cheng Huang and Hongmei Wang. A novel key-frames se- lection framework for comprehensive video summarization. IEEE Transactions on Circuits and Systems for Video Tech- nology, 30:577–589, 2019
2019
-
[15]
Chat-univi: Unified visual representation em- powers large language models with image and video under- standing
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. InCVPR, 2024
2024
-
[16]
Kuehne, H
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, , and T. Serre. Hmdb: A large video database for human motion recogni tion. InICCV, 2011
2011
-
[17]
Llava-onevision: Easy visual task transfer
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. InarXiv preprint arXiv:2408.03326, 2024
2024 arXiv
-
[18]
Mvbench: Acomprehensive multi-modal video understanding benchmark
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: Acomprehensive multi-modal video understanding benchmark. InCVPR, 2024
2024
-
[19]
Redundancy-aware transformer for video question answering
Yicong Li, Xun Yang, An Zhang, Chun Feng, Xiang Wang, and Tat-Seng Chua. Redundancy-aware transformer for video question answering. InACM MultiMedia, 2023
2023
-
[20]
Ia-red2: Interpretability-aware redundancy reduction for vision trans- formers
Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-red2: Interpretability-aware redundancy reduction for vision trans- formers. InNeurIPS, 2021
2021
-
[21]
Anderson, and Lubomir Bourdev
Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G. Anderson, and Lubomir Bourdev. Learned video compression. InICCV, 2019
2019
-
[22]
Active learning for con volutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese. Active learning for con volutional neural networks: A core-set approach. InICLR, 2018
2018
-
[23]
Steps: Self-supervised key step extrac- tion and localization from unlabeled procedural videos
Anshul Shah, Benjamin Lundell, Harpreet Sawhney, and Rama Chellappa1. Steps: Self-supervised key step extrac- tion and localization from unlabeled procedural videos. In ICCV, 2023
2023
-
[24]
Moviechat: From dense token to sparse memory for long video under- standing
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tianbo Ye, Yang Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video under- standing. InCVPR, 2024
2024
-
[25]
Ucf101: A dataset of 101 human actions classes from videos in the wild
Khurram Soomro, Amir Roshan Zamir, , and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. InCoRR, 2012
2012
-
[26]
Omnivec2- a novel transformer based network for large scale multimodal and multitask learning
Siddharth Srivastava and Gaurav Sharma. Omnivec2- a novel transformer based network for large scale multimodal and multitask learning. InCVPR, 2024
2024
-
[27]
D4m: Dataset distillation via disentangled diffusion model
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D4m: Dataset distillation via disentangled diffusion model. InCVPR, 2024
2024
-
[28]
On the diversity and realism of distilled dataset: Anefficient dataset distilla- tion paradigm
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: Anefficient dataset distilla- tion paradigm. InCVPR, 2024
2024
-
[29]
Richard S. Sutton. Learning to predict by the methods of temporal differences. InMachine-mediated learning, 1988
1988
-
[30]
Sutton and Andrew G
Richard S. Sutton and Andrew G. Barto. Time derivative models of pavlovian reinforcement. 1990
1990
-
[31]
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InICCV, 2015. 9
2015
-
[32]
Cafe: Learning to condense dataset by aligning features
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Wang Shuo Yang, Guan Huang, Hakan Bilen, Xinchao Wang, , and Yang You. Cafe: Learning to condense dataset by aligning features. InCVPR, 2022
2022
-
[34]
Dataset distillation
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, , and Alexei A Efros. Dataset distillation. InarXiv preprint arXiv:1811.10959, 2018
2018 arXiv
-
[35]
Dynamic face video segmentation via reinforcement learning
Yujiang Wang, Mingzhi Dong, Jie Shen, Yang Wu, Shiyang Cheng, and Maja Pantic. Dynamic face video segmentation via reinforcement learning. InCVPR, 2020
2020
-
[36]
Adaptive focus for efficient video recognition
Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. InICCV, 2021
2021
-
[37]
Intern- video2: Scaling foundation models for multimodal video un- derstanding
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Intern- video2: Scaling foundation models for multimodal vide...
2024
-
[38]
Dancing with still images: Video distillation via static-dynamic dis- entanglement
Ziyu Wang, Yue Xu, Cewu Lu, and Yong-Lu Li. Dancing with still images: Video distillation via static-dynamic dis- entanglement. InCVPR, 2024
2024
-
[39]
Christopher J.C.H. Watkins. Learning from delayed rewards. InPh.D. Thesis, King’s College, 1989
1989
-
[40]
Watkins and Peter Dayan
Christopher J.C.H. Watkins and Peter Dayan. Technical note q-learning. InMachine Learning, 8, 279-292, 1992
1992
-
[41]
Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S. Davis. Adaframe: Adaptive frame selection for fast video recognition. InCVPR, 2019
2019
-
[42]
Squeeze and recover and relabel: Dataset condensation at imagenet scale from a new perspective
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze and recover and relabel: Dataset condensation at imagenet scale from a new perspective. InNeurIPS, 2023
2023
-
[43]
Dataset condensation via generative model
David Junhao Zhang, Heng Wang, Chuhui Xue, Rui Yan, Wenqing Zhang, Song Bai, and Mike Zheng Shou. Dataset condensation via generative model. InarXiv preprint arXiv:2303.04707, 2023
2023 arXiv
-
[44]
Video-llama an instruction-tuned audio-visual language model for video un- derstanding
Hang Zhang, Xin Li, and Lidong Bing. Video-llama an instruction-tuned audio-visual language model for video un- derstanding. InCVPR, 2024
2024
-
[45]
Neural rate control for learned video compression
Yiwei Zhang, Guo Lu, Yunuo Chen, Shen Wang, Yibo Shi, Jing Wang, and Li Song. Neural rate control for learned video compression. InICCV, 2019
2019
-
[46]
Dataset condensation with distri- bution matching
Bo Zhao and Hakan Bilen. Dataset condensation with distri- bution matching. InWACV, 2023
2023
-
[47]
Dataset condensation with gradient matching
Bo Zhao, Konda Reddy Mopuri, , and Hakan Bilen. Dataset condensation with gradient matching. InICLR, 2021
2021
-
[48]
Im- proved distribution matching for dataset condensation
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Im- proved distribution matching for dataset condensation. In CVPR, 2023
2023
-
[49]
Cen- terclip: Token clustering for efficient text-video retrieval
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Cen- terclip: Token clustering for efficient text-video retrieval. In SIGIR, 2022
2022
-
[50]
Dataset distillation using neural feature regression
Yongchao Zhou, Ehsan Nezhadarya, , and Jimmy Ba. Dataset distillation using neural feature regression. In NeurIPS, 2022. 10
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.