{"id":"c63049e3-6e07-4f31-82ac-dee0600538f6","arxiv_id":"2504.17132","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Latent-space encoding with DPP selection and HOSVD compression outperforms prior video dataset distillation methods on MiniUCF, HMDB51, Kinetics-400, and SSv2 at IPC 1 and 5.","lead":"The authors distill video datasets by encoding clips into a pretrained autoencoder's latent space, selecting a diverse subset with determinantal point processes, and compressing those codes with tensor decomposition. The result is a small, storage-efficient dataset on which a video classifier trains to higher accuracy than prior pixel-space distillation methods on four benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Supplementary B.3 says IPC-1 distills 24 selected videos per class, not 1; the headline gains may reflect instance count, not distillation quality.","rationale":"The reader's weakest_assumption focuses on whether the VAE latent space and HOSVD truncation preserve class-discriminative information. That is a reasonable concern, but the more load-bearing issue is the comparison protocol itself: Supplementary B.3 explicitly states that 24 instances per class are selected under the IPC-1 storage budget. Even if the VAE and HOSVD preserve all relevant information, the reported gains cannot be attributed to better distillation unless the effective number of training instances per class is matched. This is not an ad hominem or an outside-consensus disagreement; it relies on the paper's own description of its pipeline. A concrete check can settle the issue by counting decoded instances per class in the released code. If the count is indeed 24, the authors must either rerun all comparisons under a true one-instance-per-class constraint or reframe the contribution as storage-budget-constrained coreset selection and compare against baselines that are also allowed multiple instances under the same budget. The reader's CONDITIONAL verdict still seems appropriate, but the condition should be sharpened to require this verification and, if necessary, a re-analysis of the headline numbers.","tokens_in":14589,"tokens_out":7445,"duration_ms":76024,"concrete_test":"Inspect the released code/configs and count the number of decoded video instances per class that are actually fed to C3D for the MiniUCF IPC-1 row of Table 1. If that count is 24, rerun the same pipeline with exactly one latent instance per class (e.g., DPP-select a single video per class, or truncate the sample mode of the HOSVD tensor to one slice) and compare accuracy with IDTD's 22.5%. If accuracy collapses to near the baseline, the reported state-of-the-art result is an artifact of instance count rather than of latent-space distillation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim ('new state-of-the-art ... outperforming prior methods on all datasets') is only meaningful if Ours and baselines are compared at the same IPC. In standard dataset distillation, IPC 1 means one distilled instance per class, and the pixel-space baselines in Table 1 are evaluated on that setting. Supplementary B.3, however, states that for MiniUCF IPC 1 'we sample 24 instances per class' before applying HOSVD at ratio 0.75; the resulting tensors occupy 27MB and the quantized VAE 80MB, for 107MB total, within the 115MB budget of a single IPC. The decoded training data therefore appears to contain roughly 24 reconstructed videos per class (about 1200 videos for MiniUCF) rather than 1 per class (about 50 videos). Matching only byte storage is not equivalent to matching IPC: the effective number of training examples, the diversity available to the learner, and the training compute all change. The 12.3-point gain on MiniUCF IPC 1 and 2.6-point gain on HMDB51 IPC 1 could be produced simply by packing more independent source videos into the budget. The introduction even frames this as 'denser instance packing under fixed storage limits.' This is an internally evidenced fairness problem, not a speculation about latent-space information loss: it directly undermines the headline SOTA claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a video dataset distillation method that operates in the latent space of a pretrained variational autoencoder. Input videos are encoded into latents, a determinantal point process (DPP) selects a diverse subset of latent codes, and high-order singular value decomposition (HOSVD) compresses the selected latent tensor. The VAE is post-training quantized and its size is included in the storage budget. The distilled latent set is decoded back to pixels to train a C3D classifier. The paper reports results on MiniUCF, HMDB51, Kinetics-400, and SSv2 under IPC 1 and IPC 5, claims state-of-the-art accuracy over pixel-space baselines including VDSD and IDTD, and provides ablations on sampling strategy, compression ratio, latent compression technique, and cross-architecture generalization.","tokens_in":14848,"tokens_out":6367,"duration_ms":56481,"significance":"If the central claim were established under matched experimental conditions, the idea of storing a compressed latent set plus a compact VAE would be a practically useful and training-free alternative to pixel-space video distillation. The paper has tangible strengths: the compression pipeline is simple and training-free, storage accounting explicitly includes the VAE model, cross-architecture evaluation is reported, and code is released. However, the headline IPC comparison is not matched in the number of reconstructed training instances, so the claimed state-of-the-art is not currently supported. The empirical contribution is therefore conditional on a substantially revised experimental protocol.","major_comments":[{"comment":"The central claim of a new state-of-the-art is not established because Instance Per Class is not actually matched. Supplementary B.3 states that for MiniUCF IPC 1 the method samples 24 instances per class before HOSVD, consuming 27MB for the distilled tensors and 80MB for the quantized 2D-VAE, within the 115MB budget. The baselines in Table 1 are evaluated with one distilled instance per class. The decoded training set therefore contains roughly 24 reconstructed videos per class (about 1200 videos for MiniUCF) rather than one per class (about 50 videos), so the large gains, especially the 12.3-point improvement on MiniUCF IPC 1, could be caused by a roughly 24-fold increase in effective training instances rather than by latent-space distillation. Matching byte storage alone is not equivalent to matching IPC. The authors should either compare under the same number of reconstructed instances per class with the same storage budget, or explicitly reframe the contribution as fixed-budget latent coreset selection/compression and compare against coreset baselines under exactly the same instance counts and storage. As written, the abstract's claim that the method outperforms prior methods on all datasets is unsupported.","section":"Supplementary B.3, Table 1"},{"comment":"The main hyperparameters appear to be selected using the evaluation datasets, which makes the reported numbers optimistic. The rank compression ratio r is set to 0.75 based on Table 3, which reports accuracy on MiniUCF and HMDB51 at different r values, and the VAE choice changes per setting: a 2D VAE (SD-VAE-FT-MSE) is used for IPC 1 and a 3D VAE (CV-VAE) for IPC 5, as described in Section B.1. No validation split or selection protocol is described. Additionally, Table 3 is non-monotonic: MiniUCF accuracy at r=1.0 is 28.9%, substantially lower than the 34.8% at r=0.75, so the selected ratio is not in a stable plateau. Please report a validation-based selection procedure or demonstrate that the main conclusions are robust over a range of r and across both VAE choices.","section":"§4.5, Table 3; §B.1"},{"comment":"The comparison between truncated SVD and HOSVD is described as being under the same storage budget, but the actual storage sizes or rank/truncation settings for each method are not reported. Without knowing the byte counts for the SVD and HOSVD representations, the reported gains (+2.6% on MiniUCF, +1.8% on HMDB51, +1.4% on Kinetics-400, +1.2% on SSv2) could reflect different compression levels rather than an advantage of preserving tensor structure. Please specify the storage consumption for each entry or fix the budget explicitly.","section":"§4.5, Table 4"}],"minor_comments":[{"comment":"The phrase 'reducing the bid-width from 32 to 8 bits' should read 'bit-width'.","section":"Appendix A.1"},{"comment":"Table 6 and the surrounding text in Section 4.5 state that the SSv2 comparison is at IPC 1, but the reported value for Ours (10.5%) matches the IPC 5 result in Table 1 rather than the IPC 1 result (6.9%). The caption appears to be mislabeled and should be corrected.","section":"Table 6 and Figure 2"},{"comment":"The notation 'V AE', 'CV-V AE', and 'SD-V AE-FT-MSE' is inconsistent; the spacing and hyphenation should be unified.","section":"Throughout"},{"comment":"The word 'distilation' in the Table 6 caption is a typo and should be 'distillation'.","section":"Table 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the IPC mismatch documented in Supplementary B.3: the method packs 24 source instances per class into the IPC 1 budget, while baselines use one distilled instance per class. This directly undermines the headline SOTA claim. If the authors cannot provide matched-instance-count comparisons or a clear reframing as fixed-budget coreset selection, the manuscript should not be accepted. The hyperparameter selection on test data and the SSv2 IPC labeling inconsistency also require correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper should have been a solid coreset paper; right now it is an apples-to-oranges distillation claim. The latent-space pipeline — VAE encoding, DPP selection, HOSVD compression — is training-free, fast, and storage-efficient, and the ablations are informative. But the IPC 1 results in Table 1 are not what they appear. Per the supplement, at MiniUCF IPC 1 the authors sample 24 videos per class and pack their compressed latent codes into the same 115MB budget that pixel-space baselines use for one synthetic video per class. So the comparison is really 24 real videos per class versus one synthetic video per class. The big gains likely come from that extra data, not from a distillation breakthrough.\n\nThere are smaller problems too. The VAE switches between 2D and 3D architectures depending on IPC; the rank compression ratio r=0.75 is tuned on the evaluation datasets; and Table 3 shows accuracy dropping at r=1.0 (less compression) without explanation. None of these are fatal alone, but they compound the main fairness issue.\n\nWhat is actually new here is the combination of latent encoding with DPP selection and HOSVD compression as a storage-efficient way to build a training set of reconstructed videos. The method itself is simple, effective, and worth building on. If the authors reframed the paper as a latent-space coreset technique under a fixed byte budget and compared against coreset baselines that also use multiple real videos per class, it would be a good contribution. As an 'IPC 1 dataset distillation SOTA' claim, it is not credible.\n\nI would send this to peer review, but demand a major revision: report the effective number of training instances per class in every experiment, compare at equal instance counts, and explain the non-monotonic compression curve. The core idea deserves referee time, but the numbers as presented are misleading.\n\nBest,","headline":"Useful latent-space coreset idea, but the SOTA claim at IPC 1 is misleading because the method packs 24 real videos per class into the same byte budget.","tokens_in":15403,"tokens_out":6300,"would_cite":false,"duration_ms":52879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Latent-space video distillation beats pixel-space methods on all four benchmarks","keywords":["video dataset distillation","latent space","variational autoencoder","determinantal point processes","high-order singular value decomposition","action recognition","dataset compression","training-free compression"],"falsifier":"Run the same DPP plus HOSVD pipeline on a motion-critical benchmark, for example Something-Something V2, but replace the pretrained VAE encoder with a random linear projection to the same latent dimension; if action-recognition accuracy stays close to the VAE-based result, the claim that the learned latent space carries the gain is false, and if accuracy collapses, the claim is supported. A second check is to vary the HOSVD temporal-mode rank on a dataset where classes differ mainly by motion and look for a sharp accuracy cliff before the reported $r=0.75$ operating point.","tokens_in":14360,"feed_emoji":"🎬","tokens_out":6002,"duration_ms":51795,"temperature":0.7,"pith_summary":"This paper tries to establish that video dataset distillation should be done in the latent space of a pretrained variational autoencoder rather than by synthesizing or selecting pixels. The proposed pipeline encodes videos into compact latent codes, picks a diverse subset with determinantal point processes, and compresses the selected codes with high-order singular value decomposition, all without training a distillation network. The authors report that this training-free pipeline outperforms every pixel-space baseline on MiniUCF, HMDB51, Kinetics-400, and Something-Something V2 at one and five videos per class, under matched storage budgets. If correct, the result matters because it makes video dataset compression far cheaper and faster while improving downstream action-recognition accuracy.","feed_headline":"Latent-space video distillation beats pixel methods on four benchmarks","feed_subtitle":"A training-free VAE, DPP, and HOSVD pipeline compresses video datasets and sets new state-of-the-art accuracy.","key_machinery":"The load-bearing object is the latent code produced by a pretrained video variational autoencoder, treated as a structured tensor rather than a flat vector; the paper uses a 3D VAE for IPC 5 and a 2D VAE for IPC 1. Three mechanisms act on it. Determinantal point processes, a probabilistic model that favors subsets with low pairwise similarity, select a diverse and representative set of latent trajectories. High-order singular value decomposition (HOSVD) factorizes the selected latent tensor into a core tensor and per-mode orthonormal factor matrices, allowing rank truncation that discards low-energy spatial and temporal components without retraining. Two-stage post-training quantization, INT8 for fully connected layers and FP16 for convolutional layers, shrinks the VAE model itself so that the stored VAE plus distilled tensors fit inside the same storage budget as pixel-space baselines.","core_discovery":"The central claim is that a training-free latent-space pipeline can replace expensive pixel-space synthesis in video dataset distillation. The paper argues that a pretrained 3D variational autoencoder already captures the spatiotemporal structure needed for action recognition, so the distillation problem reduces to selecting a diverse subset of latent trajectories and compressing them. DPP-based selection chooses samples that are both representative and well-spread in the latent space, while HOSVD truncation removes low-energy components in each tensor mode, including the temporal mode, to pack more information into a fixed storage budget. The paper reports new state-of-the-art results on all four datasets, with the largest gains at the most extreme compression: 34.8% versus 22.5% on MiniUCF at one instance per class.","pith_inferences":["Editorial: If the VAE latent space really carries the discriminative signal, the same encode-select-compress recipe could transfer to other data modalities that have strong pretrained autoencoders, such as audio or medical imaging, where pixel-space distillation would be even more expensive.","Editorial: The training-free nature means the compute advantage should grow with dataset size; a natural testable extension is applying the pipeline at a scale where iterative pixel-space distillation becomes impractical.","Editorial: HOSVD truncation in the temporal mode acts as an implicit motion summarizer, so the rank ratio could potentially be tuned per class or per motion type rather than globally, which the paper does not explore.","Editorial: The storage budget includes the quantized VAE model itself, so the method's advantage may depend on how far the VAE cost is amortized; at higher IPC settings the breakeven point relative to pixel-space baselines deserves scrutiny."],"forward_implications":["Under matched storage budgets, the method reports higher accuracy than all pixel-space baselines on MiniUCF, HMDB51, Kinetics-400, and SSv2 at both IPC 1 and IPC 5.","Because the pipeline is training-free, distillation runtime drops from hours to minutes on small datasets and to about one hour on Kinetics-400 and SSv2, versus over five hours for the VDSD baseline.","Distilled latent datasets transfer across evaluation architectures, including ConvNet3D, CNN+GRU, and CNN+LSTM, not just the architecture used during selection.","The HOSVD rank ratio controls the accuracy-storage tradeoff: $r=0.75$ is the reported sweet spot, while $r=0.1$ degrades accuracy to near-random levels.","Quantizing the VAE achieves more than a 2.6x model compression ratio while retaining reconstruction fidelity, which is what lets the total stored package remain within the IPC storage budget."],"supporting_citations":[{"why":"Supplies the 3D VAE whose latent trajectories are the representation the whole pipeline operates on.","marker":"[47]"},{"why":"Supplies the determinantal point process framework used to select a diverse and representative latent subset.","marker":"[17]"},{"why":"The VDSD pixel-space baseline and prior static-dynamic disentanglement method that the paper compares against and extends.","marker":"[39]"},{"why":"The IDTD baseline, the strongest prior video distillation method the paper reports outperforming on all settings.","marker":"[48]"},{"why":"Prior work on latent image dataset distillation that motivates applying latent-space distillation to video.","marker":"[25]"},{"why":"Supplies the 2D VAE used at IPC 1 to fit the stricter storage budget in the supplementary fair-comparison analysis.","marker":"[1]"},{"why":"The distribution-matching baseline used in main comparisons and in the inter-frame visualization contrast.","marker":"[45]"}],"fun_headline_variants":["Latent-space video distillation sets new SOTA on all benchmarks","Training-free latent pipeline tops video dataset distillation","Latent video distillation: training-free, state-of-the-art","Video distillation goes latent: better with less data","Latent-space trick shrinks video datasets, boosts accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a pretrained variational autoencoder, a network trained to compress videos into a compact latent space and reconstruct them, keeps the class-discriminating motion and appearance information even after high-order SVD truncation, even though it was trained for reconstruction, not for classification. If truncation removes class-relevant structure, the accuracy gain collapses.","fun_headline_variants_meta":{"raw":{"variants":["Latent-space video distillation sets new SOTA on all benchmarks","Training-free latent pipeline tops video dataset distillation","Latent video distillation: training-free, state-of-the-art","Video distillation goes latent: better with less data","Latent-space trick shrinks video datasets, boosts accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3535,"prompt_tokens":884,"completion_tokens":2651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":2573}},"tokens_in":500,"tokens_out":2651,"duration_ms":17931,"temperature":1.0,"reasoning_tokens":2573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:47:50.622312+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same DPP plus HOSVD pipeline on a motion-critical benchmark, for example Something-Something V2, but replace the pretrained VAE encoder with a random linear projection to the same latent dimension; if action-recognition accuracy stays close to the VAE-based result, the claim that the learned latent space carries the gain is false, and if accuracy collapses, the claim is supported. A second check is to vary the HOSVD temporal-mode rank on a dataset where classes differ mainly by motion and look for a sharp accuracy cliff before the reported $r=0.75$ operating point.","supporting_citations":[{"cited_title":"Cv- vae: A compatible video vae for latent generative video mod- els","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D VAE whose latent trajectories are the representation the whole pipeline operates on."},{"cited_title":"Determinantal point processes for machine learning","cited_arxiv_id":null,"evidence_quote":"Supplies the determinantal point process framework used to select a diverse and representative latent subset."},{"cited_title":"Dancing with still images: Video distillation via static-dynamic dis- entanglement","cited_arxiv_id":null,"evidence_quote":"The VDSD pixel-space baseline and prior static-dynamic disentanglement method that the paper compares against and extends."},{"cited_title":"https : / / huggingface","cited_arxiv_id":null,"evidence_quote":"Supplies the 2D VAE used at IPC 1 to fit the stricter storage budget in the supplementary fair-comparison analysis."},{"cited_title":"Dataset condensation with distri- bution matching","cited_arxiv_id":null,"evidence_quote":"The distribution-matching baseline used in main comparisons and in the inter-frame visualization contrast."}],"review_version":1}