{"id":"68f9a115-f441-4d7f-a1da-e55f9d49507e","arxiv_id":"2412.00111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IDTD distills a video dataset into a small set of synthetic videos by jointly reducing redundancy between videos and inside each video, and it reports accuracy gains over prior video dataset distillation baselines on several action-recognition benchmarks.","lead":"This paper introduces IDTD, a method that shrinks large video datasets by creating a small set of synthetic videos that preserve both the variety between videos and the useful motion inside each video. If it works, it could make training video-based AI models cheaper and faster on large action-recognition collections.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own notation implies per-instance Feature Pools, so the claimed shared-pool mechanism for inter-sample diversification may not exist; the SOTA results may come from a different mechanism.","rationale":"The reader's weakest assumption is the lack of cross-architecture transfer, which is a valid external-validity concern: the matching loss uses ConvNet3D features, so the synthetic videos may be overfit to that architecture and fail to transfer to other backbones. However, I see an even more fundamental internal inconsistency that directly threatens the central claim about jointly reducing both redundancy dimensions. The notation and pseudocode in Section 3.2 and Algorithm 1 assign a distinct Feature Pool to each synthetic instance, while the surrounding text claims a shared per-class pool that preserves common class knowledge. The diversity loss only operates within one instance, so the architecture as written does not explicitly diversify across instances. This means the inter-sample redundancy reduction, a core contribution, may not be implemented at all. This is not an external critique about generalization; it is about whether the method does what it claims. If the implementation actually uses per-instance pools, the paper's conceptual contribution is severely weakened: the method reduces to independent per-instance distillation plus within-video diversity, and the 'joint' inter-sample aspect is unsupported. The empirical results could still be valid, but they would not support the stated mechanism. The concrete test would settle the issue by comparing the two pool-sharing schemes. Until then, the paper should not be accepted as-is, but it is also not clearly rejectable because the empirical results and within-sample mechanism may survive. The reader's verdict of CONDITIONAL remains appropriate, though for a different reason than the one emphasized in the reader's weakest_assumption.","tokens_in":10419,"tokens_out":5796,"duration_ms":54780,"concrete_test":"Run the training exactly as described with two variants: (a) a single per-class Feature Pool shared across all M synthetic instances, as the prose claims, and (b) per-instance Feature Pools as implied by Algorithm 1. Compare MiniUCF IPC=5 accuracy under identical budgets. If (a) and (b) are statistically indistinguishable, the inter-sample mechanism is not the source of the gains and the joint-redundancy claim is unsupported. If (a) is significantly worse, the paper's description is wrong and the method should be re-scoped. Also check the released code or request tensor shapes of P and S to settle which structure was actually implemented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that IDTD jointly reduces within-sample and inter-sample redundancy, with a shared per-class Feature Pool preserving common class knowledge. However, Section 3.2 formally defines P = {{p_{m,n}}_M}_N and S = {{{s_{m,n,k}}_K}_M}_N, and Algorithm 1 generates d_{m,n,k} = s_{m,n,k}(p_{m,n}). This means each synthetic instance m has its own Feature Pool p_{m,n}; the pool is shared only among the K Feature Selectors of that same instance, not across instances of a class. The diversity loss in Eq. (1) computes L2 distance between segments d_{m,n,k} and d_{m,n,q} belonging to the same instance, so it enforces within-sample diversity only. There is no loss term or shared variable that explicitly diversifies different synthetic instances m and m'. Consequently, the described architecture does not explicitly reduce inter-sample redundancy; it reduces within-sample redundancy and then independently optimizes each synthetic video. If the implementation matches the notation, the claimed joint inter-sample diversification mechanism is absent, and the reported gains over DM+VDSD may instead arise from per-instance matching plus within-video diversity. If the authors intended a per-class shared pool, the paper's notation and algorithm are inconsistent with that intent and need correction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Video Set Distillation as a new problem and proposes IDTD (Information Diversification and Temporal Densification), a method that synthesizes a compact set of videos from a real video dataset. The method uses per-instance Feature Pools and Feature Selectors to generate K diverse segments, a Temporal Fusor to combine them into a synthetic video, and a Stochastic Temporal Augmentation during training. The overall objective combines a dataset distillation matching loss on the segments and on the fused video with a diversity loss on the segments. The authors claim that IDTD jointly reduces within-sample and inter-sample redundancy and report state-of-the-art results on MiniUCF, HMDB51, SSv2, and K400, with ablations and qualitative results.","tokens_in":10721,"tokens_out":7890,"duration_ms":71069,"significance":"If the central claim is supported, this would be a valuable new direction for dataset distillation: it explicitly targets the two-layer nested redundancy of video sets, which prior image-level distillation methods and VDSD do not jointly address. The paper provides a modular design, experiments on four datasets, and ablations that isolate the contributions of the modules and losses. The empirical gains at higher IPC on the small-scale datasets are encouraging. However, the formal description of the method raises a load-bearing concern: as written, the mechanism does not actually minimize inter-sample redundancy, and the reported tables contradict the unqualified state-of-the-art claim in the abstract. These issues need to be resolved before the paper's main contribution can be accepted.","major_comments":[{"comment":"The formal definition P = {{p_{m,n}}_M}_N and S = {{{s_{m,n,k}}_K}_M}_N, together with the update rule d_{m,n,k} = s_{m,n,k}(p_{m,n}) in Algorithm 1, describes a separate Feature Pool p_{m,n} for each synthetic instance m of class n, not a per-class shared pool. The diversity loss in Eq. (1) compares only segments within the same instance (same m and n), so no term in the objective promotes diversity across different synthetic instances m and m' or shares information across them. Consequently, the claimed joint minimization of inter-sample redundancy is not realized by the written method; as written, the method reduces within-sample redundancy and then independently optimizes each synthetic video. The authors need to either change the formulation to a per-class shared pool and add an explicit inter-instance diversity mechanism, or substantiate with analysis and experiments how the per-instance formulation reduces inter-sample redundancy. This issue is load-bearing for the paper's central claim.","section":"Section 3.2 and Algorithm 1"},{"comment":"The training objective L = L_M((D, T) + alpha1 * L_div + alpha2 * L_M((V'_syn, T) has unbalanced parentheses and cannot be parsed as written. In addition, L_M is never concretely defined; the text only says it is a 'dataset distillation matching objective function between synthetic and real videos'. The sentence immediately after Eq. (2) also says 'within-sample redundancy and inter-class redundancy are jointly reduced', which should be 'inter-sample redundancy'. Please provide a well-formed objective and a precise definition of L_M, since this equation is central to the method and to reproducibility.","section":"Section 3.4, Eq. (2)"},{"comment":"The abstract states that the method 'achieves state-of-the-art results in Video Dataset Distillation', but the reported numbers contradict this at IPC=1. On MiniUCF, MTT+VDSD achieves 23.3±0.6 while IDTD achieves 22.52±0.1; on K400, MTT+VDSD and DM+VDSD achieve 6.3±0.1 and 6.3±0.2 while IDTD achieves 6.1±0.1; on SSv2, MTT+VDSD achieves 5.5±0.1 while IDTD achieves 3.9±0.1. The SOTA claim must be qualified to the settings where it actually holds (higher IPC), or the text must explain why these IPC=1 cases are excluded from the claim.","section":"Table 1 and Table 2"}],"minor_comments":[{"comment":"The upper limit of the outer sum is K-2, which for K segments omits the pair (K-1, K); the limit should presumably be K-1. The notation also does not show m and n indices on L_div, which makes it less clear that the loss is applied per instance.","section":"Eq. (1)"},{"comment":"Line 16 says 'Calculate loss by equation 1', but Eq. (1) is only the diversity loss; the update should use the total loss from Eq. (2). Also, Algorithm 1 calls tau(v_{m,n}) while Section 3.3 defines tau(v_{m,n}, mu); the two formulations should be consistent.","section":"Algorithm 1"},{"comment":"The caption cites 'SOTA [37]', but the reference list ends at [26]; the citation either needs a corresponding reference or should be removed.","section":"Figure 3"},{"comment":"There are several typos, including 'Kenetics400' (should be Kinetics-400), 'Diversificaiton' in the Figure 2 caption, and 'optimzition' in Section 4.5.","section":"Section 4.1"},{"comment":"No cross-architecture transfer experiment is reported: the student model in the matching loss and the evaluated model are both ConvNet3D. An experiment that trains a different video architecture on the same synthetic set would strengthen the claim that the synthetic videos are generally useful, but its absence is not fatal for the same-architecture comparison.","section":"Section 4.2 and evaluation"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study, but the core conceptual claim needs to be checked against the actual implementation. The notation and algorithm strongly suggest per-instance feature pools, which would mean the 'inter-sample diversification' mechanism is not present in the formalization; if the implementation indeed uses per-class shared pools, the paper needs to correct the notation and provide an explicit mechanism or evidence for inter-sample diversification. The SOTA claim should also be qualified to match the tables. These issues are substantial but fixable within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper's central claim doesn't survive its own equations. The abstract promises joint within-sample and inter-sample redundancy reduction, but the formal setup defines a separate Feature Pool for each synthetic instance, and the diversity loss only separates segments within the same video. There is no term that pushes different synthetic videos apart. So the 'inter-sample' part of the contribution is not actually implemented.\n\nWhat's new? The specific architecture—per-instance Feature Pool, K linear Feature Selectors, a diversity loss, and a Temporal Fusor with stochastic temporal augmentation—is not in prior video distillation work. The ablations in Tables 3 and 4 are the most informative part: they show each loss and module contributes, and the temporal-size curve in Fig. 3 is a genuine effect. At IPC=5 the numbers beat DM+VDSD and MTT+VDSD on MiniUCF, HMDB51, K400, SSv2, and that's a real result even if not universal.\n\nThe flaws are proportionate but real. First, the 'first to study Video Set Distillation' claim is false; VDSD is exactly video set distillation. Second, Table 2 shows IDTD underperforms several baselines at IPC=1 on K400 and SSv2, yet the abstract says SOTA everywhere; the paper even admits 'only comparable' at low IPC. Third, no code or cross-architecture transfer; the method is only evaluated with ConvNet3D both as feature extractor and as student, which leaves open that the gains are tied to that feature space. Finally, there are small technical errors: Eq. (1) summation bound, malformed parentheses in the loss, and the text says 'inter-class' where it means 'inter-sample'.\n\nWho is this for? Someone working on dataset distillation for video might read the ablation and the temporal densification idea. The core mechanism, however, does not match the paper's premise, so the result is a better within-video diversity method, not a joint framework. That's still worth a referee's time if the authors fix the claims and the formalism, but as written the central narrative is misaligned.\n\nRecommendation: I'd send it to review with a request for major revision, mainly to reconcile the notation with the claimed mechanism, and to restrict the claims to what the tables support. But I wouldn't cite it in its current form.","headline":"The paper's claimed inter-sample diversification mechanism is not present in its own formalism; what remains is a within-video diversity method with solid gains at higher IPC but overclaimed SOTA.","tokens_in":11249,"tokens_out":4333,"would_cite":false,"duration_ms":34291,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims IDTD is the first video-set distillation method to cut both within-clip and between-clip redundancy at once, reporting state-of-the-art accuracy on four action datasets.","keywords":["video set distillation","dataset distillation","temporal redundancy","information diversity","feature pool","temporal fusor","video recognition","synthetic video data"],"falsifier":"Train a video classifier with a different architecture (for example, a video transformer or a different 3D CNN) on the synthetic set produced by IDTD and evaluate on the same test split: if accuracy collapses to near random while the same synthetic set trains ConvNet3D well, the distilled set encodes extractor-specific features rather than general temporal action information. A cheaper check is to remove Stochastic Temporal Augmentation and retrain; a large drop would indicate the augmentation, not the synthetic content, is responsible for the reported accuracy.","tokens_in":10204,"feed_emoji":"🎬","tokens_out":11422,"duration_ms":85784,"temperature":0.7,"pith_summary":"The paper introduces Video Set Distillation, a new problem: generating a small synthetic video set that can train a recognition model efficiently while preserving the temporal information needed for action recognition. It argues that video sets carry two nested redundancy dimensions—similar frames within a clip and similar clips within a class—and that key-frame selection, dataset pruning, and image-level dataset distillation each remove only one of them. IDTD is the proposed method, jointly optimizing both dimensions through a shared Feature Pool with K diverse Feature Selectors and a Temporal Fusor that packs diverse segments into one synthetic video. The paper reports state-of-the-art accuracy on MiniUCF, HMDB51, SSv2, and Kinetics-400, with the largest gains on classes where both redundancies are high. If the claim holds, large video datasets could be replaced by much smaller synthetic sets for efficient model training.","feed_headline":"IDTD distills video datasets by cutting both redundancy dimensions","feed_subtitle":"IDTD reports top accuracy on four video-distillation benchmarks; gains grow with redundancy.","key_machinery":"The load-bearing mechanism is the division of labor between a shared per-class Feature Pool and K feature-specific Selectors, supervised by two competing losses. Each Selector is a lightweight linear layer that transforms the pool into a Diverse Segment, and the diversity loss $\\mathcal{L}_{div}$ maximizes pairwise $L_2$ distances between segment features, so the Selectors spread out while the pool preserves class-common knowledge. The Temporal Fusor is a convolution along time that fuses the K segments into one video $v_{m,n} = f_{m,n}(\\{d_{m,n,k}\\})$, and Stochastic Temporal Augmentation $\\tau(\\cdot)$ randomly crops a temporal interval of $v_{m,n}$ and reshapes it to the target length, preventing fixed temporal positions and forcing the synthetic video to be temporally dense. The overall objective combines a dataset-distillation matching loss on the diverse segments, the diversity loss, and a second matching loss on the augmented synthetic video; the paper uses a ConvNet3D student model's features to compute these losses.","core_discovery":"The paper's central claim is that a video set can be distilled by jointly removing within-sample and inter-sample redundancy, and that the two must be optimized together rather than sequentially. On the author's terms, IDTD builds each synthetic video from K diverse segments: K randomly initialized linear Feature Selectors read a shared learnable Feature Pool and emit segments $d_{m,n,k} = s_{m,n,k}(p_{m,n})$; a diversity loss pushes the segment features apart in $L_2$ distance while a dataset-distillation matching loss aligns them with real data. A Temporal Fusor, a light convolution over time, stitches the segments into a single synthetic video of the target temporal size, and Stochastic Temporal Augmentation randomly samples a temporal interval and rescales it to that size during training. The paper reports that this two-part mechanism outperforms image-level distillation baselines and the static-image-based VDSD on all four evaluated datasets.","pith_inferences":["An editorial inference: the Feature Pool / Feature Selector decomposition could generalize to other nested-set data (multi-view images, point-cloud sequences) where within-sample and inter-sample redundancy co-occur; a testable extension would apply the same two-loss structure there.","An editorial inference: because evaluation is only reported on ConvNet3D, an open question is whether the synthetic videos carry class information to other architectures; a cross-architecture benchmark would settle whether the features are genuinely video-level or extractor-specific.","An editorial inference: Stochastic Temporal Augmentation resembles a temporal dropout regularizer; ablating it while fixing the rest would reveal how much of the accuracy gain comes from augmentation rather than from the synthesized content.","An editorial inference: the diversity loss operates in feature space only; adding a pixel- or trajectory-space diversity penalty might further improve transferability, but the paper does not explore this."],"forward_implications":["IDTD's synthetic videos can replace a large real video set for training recognition models, cutting storage and compute while staying competitive with or better than prior distillation outputs.","Larger Instance Per Class budgets yield larger accuracy gains, so the method becomes more attractive as the allowed synthetic set size grows.","Classes with high within-sample and inter-sample redundancy benefit most, so the method is best suited to repetitive, temporally redundant video collections.","The joint objective is compatible with existing dataset distillation matching losses, so IDTD can be layered onto different distillation objectives rather than replacing them."],"supporting_citations":[{"why":"Defines dataset distillation as the synthesis of a small representative set; IDTD extends this objective from images to video sets.","marker":"[18]"},{"why":"Prior video distillation method that distills a static image and interpolates motion; it is the main baseline IDTD aims to surpass and the source of the ConvNet3D student and evaluation setup.","marker":"[19]"},{"why":"Distribution Matching is the underlying matching loss that IDTD integrates with its diversity loss and temporal fusor; also a key comparison baseline.","marker":"[23]"},{"why":"Matching Training Trajectories is a bi-level optimization baseline whose performance is compared on all datasets.","marker":"[2]"},{"why":"Feature regression distillation baseline used in comparisons, showing IDTD's advantage over feature-alignment methods.","marker":"[26]"},{"why":"Kinetics-400 dataset used for large-scale evaluation of IDTD.","marker":"[1]"},{"why":"Something-Something v2 dataset used for large-scale evaluation.","marker":"[7]"},{"why":"HMDB51 dataset used in small-scale evaluation.","marker":"[10]"},{"why":"UCF101 dataset used to build MiniUCF, the small-scale evaluation benchmark.","marker":"[14]"}],"fun_headline_variants":["Distilling video sets: one model, two redundancy cuts","Cut both video redundancy types in one distillation","Jointly trimming within and across video redundancy","IDTD: squeezing video sets without losing info","Video set distillation that trims both redundancy dimensions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a shared per-class Feature Pool plus K random linear Feature Selectors and a diversity loss can represent the information distribution of a real video class, and that the matching and diversity losses computed with ConvNet3D features guarantee that synthetic videos will train a same-architecture student to generalize to real test data; no cross-architecture transfer is reported.","fun_headline_variants_meta":{"raw":{"variants":["Distilling video sets: one model, two redundancy cuts","Cut both video redundancy types in one distillation","Jointly trimming within and across video redundancy","IDTD: squeezing video sets without losing info","Video set distillation that trims both redundancy dimensions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2863,"prompt_tokens":972,"completion_tokens":1891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1820}},"tokens_in":588,"tokens_out":1891,"duration_ms":13803,"temperature":1.0,"reasoning_tokens":1820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:43:39.016297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a video classifier with a different architecture (for example, a video transformer or a different 3D CNN) on the synthetic set produced by IDTD and evaluate on the same test split: if accuracy collapses to near random while the same synthetic set trains ConvNet3D well, the distilled set encodes extractor-specific features rather than general temporal action information. A cheaper check is to remove Stochastic Temporal Augmentation and retrain; a large drop would indicate the augmentation, not the synthetic content, is responsible for the reported accuracy.","supporting_citations":[{"cited_title":"Dancing with still images: Video distillation via static-dynamic dis- entanglement","cited_arxiv_id":null,"evidence_quote":"Prior video distillation method that distills a static image and interpolates motion; it is the main baseline IDTD aims to surpass and the source of the ConvNet3D student and evaluation setup."},{"cited_title":"Dataset condensation with distri- bution matching","cited_arxiv_id":null,"evidence_quote":"Distribution Matching is the underlying matching loss that IDTD integrates with its diversity loss and temporal fusor; also a key comparison baseline."},{"cited_title":"Dataset distillation by matching training trajectories","cited_arxiv_id":null,"evidence_quote":"Matching Training Trajectories is a bi-level optimization baseline whose performance is compared on all datasets."},{"cited_title":"Dataset distillation using neural feature regression","cited_arxiv_id":null,"evidence_quote":"Feature regression distillation baseline used in comparisons, showing IDTD's advantage over feature-alignment methods."},{"cited_title":"Quo vadis and action recognition? a new model and the kinetics dataset","cited_arxiv_id":null,"evidence_quote":"Kinetics-400 dataset used for large-scale evaluation of IDTD."},{"cited_title":"The ”something something” video database for learning and evaluating visual common sense","cited_arxiv_id":null,"evidence_quote":"Something-Something v2 dataset used for large-scale evaluation."},{"cited_title":"Kuehne, H","cited_arxiv_id":null,"evidence_quote":"HMDB51 dataset used in small-scale evaluation."},{"cited_title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","cited_arxiv_id":null,"evidence_quote":"UCF101 dataset used to build MiniUCF, the small-scale evaluation benchmark."}],"review_version":1}