{"id":"b465c072-6e6f-4eee-9745-a17ab4ce2f80","arxiv_id":"2508.21773","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A non-parametric mean-shift clustering method over pretrained video transformer features enables unsupervised continual learning on video datasets without labels or task boundaries.","lead":"This paper introduces an unsupervised continual learning method for videos that uses kernel density estimation (mean-shift clustering) on features from a frozen video transformer, with a novelty detector that dynamically creates new clusters. The authors report strong accuracy on UCF101, HMDB51, and Something-Something V2 without using any labels or class boundaries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance comparison confounds KDE clustering with a much stronger frozen feature extractor; without same-feature clustering baselines the central claim is unproven.","rationale":"The reader identified the same load-bearing concern: the comparison between the proposed method and baselines is confounded by different feature extractors (frozen VideoMAE V2 vs. trained TSN/ResNet-34). This is the central issue because the paper's abstract claims the methodology 'substantially enhances the performance,' but the experiments do not isolate whether the improvement comes from KDE-based clustering, novelty detection, and replay, or simply from using a much stronger, frozen pretrained backbone. The lack of a same-feature control (e.g., K-means on the same VideoMAE features) means the central claim is unsupported even if the reported numbers are reproducible. The proposed concrete test would settle this by holding all other factors fixed and varying only the clustering mechanism. I also note a minor technical issue: Eq. (2) appears to have a sign error (the numerator should be +x_i, not -x_i), but since the authors use Scikit-learn's mean shift in implementation, this is likely a typo and not the main concern. Given the reader's verdict is already CONDITIONAL, my stress test does not change the verdict; it reinforces the need for the same-feature clustering baselines before acceptance.","tokens_in":15928,"tokens_out":4721,"duration_ms":53400,"concrete_test":"On UCF101 fold 1, take the same frozen VideoMAE V2 features used by uVCL-KDE-RBF, the same per-task data order, the same memory buffer of 20 features per cluster, and the same pseudo-label + Focal Loss linear classifier (Eq. 4-5). Replace the mean-shift/KDE assignment (Section 4.2) with (a) K-means with k equal to the number of classes actually present in the accumulated feature set at each task (or k chosen by silhouette), and (b) a simple incremental nearest-centroid classifier where each class prototype is the mean of features assigned to its cluster and no RBF layer is trained. Compute final CAcc and ACAcc under the Appendix D metric. If either control reaches within ~2 points of the reported 93.45/93.01, the KDE clustering is not the source of the advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states the proposed method uses VideoMAE V2 (pretrained on Kinetics-700) features of size 1024, frozen (Section 4.1), while Appendix C shows all baselines (iCaRL, iCaRL+CL, EWC, MAS) are adapted from vCLIMB using TSN with a ResNet-34 backbone that is trained/updated during continual learning. The proposed method therefore never updates its representation and cannot suffer representation-level catastrophic forgetting; the baselines can. The headline result (uVCL-KDE-RBF CAcc 93.45 on UCF101 vs iCaRL 10.23) may reflect the gap between frozen VideoMAE features and learned ResNet-34 features, not the contribution of KDE clustering, novelty detection, or memory replay. The ablations in Section 5.3 only vary memory size, thresholds, and bandwidth; there is no control with the same VideoMAE features and a standard clustering method (e.g., K-means) or even a non-clustering nearest-prototype classifier. Thus the paper does not establish that the proposed KDE-based mechanism is what 'substantially enhances' performance. The correct test is to hold features, memory budget, replay, and evaluation fixed and vary only the clustering/assignment mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised video continual learning (uVCL) setting in which neither task boundaries nor class labels are available, and introduces uVCL-KDE / uVCL-KDE-RBF, a non-parametric method that clusters frozen video features via mean-shift Kernel Density Estimation, stores per-cluster feature exemplars for replay, and uses novelty thresholds to create clusters dynamically. Experiments on UCF101, HMDB51, and Something-Something V2 report high cluster accuracy for the proposed method compared with iCaRL, iCaRL+CL, EWC, and MAS adapted from vCLIMB. The abstract claims that the proposed methodology substantially enhances performance when successively learning many tasks.","tokens_in":16232,"tokens_out":2890,"duration_ms":37036,"significance":"If the reported gains were attributable to the proposed clustering and novelty-detection mechanism, this would be a useful contribution to a relatively unexplored problem: it provides a benchmark protocol, a memory-efficient feature-replay strategy, and a non-parametric way to grow clusters without labels or task boundaries. The paper also ships an unusual amount of experimental detail (forgetting metrics, memory-size ablations, threshold sensitivity, t-SNE visualizations). However, the central empirical claim is currently not established because the proposed method and the baselines are compared under different feature-extraction regimes: the proposed method uses a frozen VideoMAE V2 transformer pretrained on Kinetics-700, while the baselines use a TSN/ResNet-34 backbone that is trained and updated during continual learning. The performance gap may therefore reflect the quality of the frozen features rather than the KDE-based clustering, replay, or novelty detection. The paper's stated contribution would be credible only after same-feature, same-memory-budget comparisons with simpler clustering baselines (e.g., K-means or nearest-prototype) are added.","major_comments":[{"comment":"The headline comparison is confounded. The proposed method extracts features with a frozen VideoMAE V2 (1024-d, pretrained on Kinetics-700) and never updates the representation, while all baselines are adapted from vCLIMB using TSN with ResNet-34 that is trained/updated during continual learning. Thus the gap (e.g., CAcc 93.45 vs. 10.23 on UCF101) cannot be attributed to the KDE clustering, novelty detection, or replay mechanism. To support the central claim, the authors must add baselines that use the same frozen VideoMAE V2 features and the same memory/replay budget, varying only the assignment/clustering mechanism (e.g., K-means, agglomerative clustering, nearest-prototype). Without such a control, the abstract's claim of 'substantially enhances' is unproven.","section":"§5.1, Table 1, Appendix C"},{"comment":"The hyperparameters h (mean-shift bandwidth) and Θ2 (novelty threshold) are selected per dataset by evaluating on the same datasets used for the final reported results. For example, §5.2 states 'the best results are obtained for Θ2 = 0.3' and h is tuned among {15,...,19}; Table 3 further tunes Θ2. This makes the reported numbers fitted optima rather than predictive evaluations. The paper should either use a held-out validation split for hyperparameter selection or report performance across the full grid and the associated variability. At minimum, error bars over the three splits should be reported; currently only averages are given, and for SSv2 only the first 30 tasks are evaluated despite the protocol describing 659 tasks.","section":"§5.2, Tables 1–3"},{"comment":"Memory content is not matched between the proposed method and the baselines. The proposed method stores only embedded features (1024-d vectors) in FIFO buffers, whereas the baselines, as adapted from vCLIMB, store video data with a much larger memory footprint, and Appendix C notes the baselines can retain up to 100% of training data. This difference in memory representation and management can affect forgetting behavior independently of the clustering method. The comparison should hold memory budget and memory content (features vs. raw videos) fixed, or justify why storing features is a fair and realistic baseline condition.","section":"Appendix A and Appendix C"},{"comment":"The novelty detection thresholds are load-bearing for cluster growth. Θ1 is estimated as the maximum distance between cluster centers from the first task, and Θ2 is a softmax confidence threshold. The ablation varies Θ2 but does not vary Θ1, and no analysis is given of how sensitive the final CAcc is to the first-task estimate of Θ1 or to the choice of distance metric. Since the method's ability to 'dynamically expand memory clusters' is a core contribution, this sensitivity should be quantified. If performance collapses for reasonable variations in Θ1, the robustness claim in the conclusion is overstated.","section":"Eq. (6)–(7), §5.3"}],"minor_comments":[{"comment":"The mean-shift update formula appears malformed: the Gaussian term in the numerator should involve exp(-||µ_t^j - x_i||^2 / (2h^2)), but the displayed equation has a stray minus sign on x_i and an ambiguous exponent. Please correct the notation.","section":"Eq. (2)"},{"comment":"The definition of Forward Forgetting is unusual: it averages (CAcc_{j-1} - CAcc_j), which measures degradation from the immediately preceding task, not the influence on all future tasks. The verbal description ('decreases the performance on previous task k-1') corresponds more naturally to backward forgetting. Please align the metric definition with the standard GEM definition or clarify the intended semantics.","section":"Appendix D, Eq. (9)"},{"comment":"There are several typos and inconsistencies: 'Backword' for backward, 'RestNet' in Table 5, 'SSv1' in the Section 5.2 header, 'assued' in Section 5.2, and inconsistent naming uVCL-KDE vs. uVCIL-KDE in Appendix F. Please proofread.","section":"Throughout"},{"comment":"The baseline description says 'each task introduces new 128 clusters' but the task sizes are 256 videos, which is internally inconsistent. Also, the baselines are re-implemented, but no code or detailed hyperparameters are provided; this limits reproducibility.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the proposed benchmark protocol is potentially useful. However, the empirical comparison as it stands is likely to be judged as confounded by any expert reader: different backbones, different memory formats, and per-dataset tuning of the key hyperparameters. The authors should be asked to provide same-feature baselines (ideally including a simple clustering method), report error bars, and clarify validation/hyperparameter selection before the central claim can be accepted. This is fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the paper gives the under-explored problem of unsupervised video continual learning a proper benchmark and a simple recipe based on mean-shift KDE clustering of frozen VideoMAE features. That part is new and worth having. What is not established is that the KDE mechanism is what produces the large gains over iCaRL, EWC, and MAS.\n\nThe strengths: the mixed-class task setting with no class labels is a real gap, and the protocol with 256-video chunks, dynamic cluster counts, novelty thresholds, and feature replay is a reasonable first target for the field. The ablations on memory size and thresholds are a start, and the t-SNE visualizations give some qualitative sense of cluster separation.\n\nThe soft spots are substantial, though. Every baseline uses TSN with a ResNet-34 backbone trained during continual learning, while the proposal uses frozen VideoMAE V2 features. So the headline gap could be mostly feature quality, not clustering. There is no control using VideoMAE features with K-means or a nearest-prototype classifier, no error bars across the three splits, no code, and the SSv2 baseline comparison stops at 30 tasks while the proposal runs all 659. Hyperparameters h and Θ2 are chosen per dataset on the same data used for reporting, so those numbers are fitted optima, not predictions. Also, the 'no task boundaries' claim is a bit loose: the protocol feeds fixed 256-sample chunks and evaluates after each chunk, so task identity is available to the experimenter even if the model is not told the class boundaries.\n\nThe math and the writing are otherwise coherent; the method is not tautological. But the empirical evidence does not yet support the abstract's claim that the KDE-based clustering 'substantially enhances' performance.\n\nI'd send this to peer review, asking the authors to add same-feature clustering baselines, report variance across splits, validate thresholds on held-out data, and release code. The problem is important enough to justify referee time, but the current paper needs those controls before the central claim is credible.","headline":"Useful problem setup and benchmark protocol, but the headline result is confounded by different backbones; same-feature clustering baselines are needed.","tokens_in":16691,"tokens_out":3775,"would_cite":false,"duration_ms":39402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video model can keep learning from unlabelled streams by treating its feature space as a density landscape, clustering with kernel density estimation and replaying stored features to avoid forgetting.","keywords":["unsupervised video continual learning","kernel density estimation","mean-shift clustering","catastrophic forgetting","memory replay","video action recognition","non-parametric clustering"],"falsifier":"Re-run the uVCL experiment on UCF101 with k-means or agglomerative clustering in place of mean-shift, using the same frozen video-transformer features, the same memory buffers of 20 features per cluster, the same 256-video mixed-class tasks, and the same RBF readout; if the cluster accuracy stays near 93%, then the KDE/mean-shift machinery is not the source of the improvement.","tokens_in":136,"feed_emoji":"🎬","tokens_out":4722,"duration_ms":192161,"temperature":0.7,"pith_summary":"The paper claims that a video model can continually learn a long sequence of unlabelled tasks—without task boundaries, class labels, or a pre-set number of classes—by maintaining a non-parametric density estimate of deep video features. It groups each task's videos through kernel density estimation and mean-shift clustering, stores a few features per discovered cluster for replay, and opens new clusters when incoming data looks novel. A linear readout version, uVCL-KDE-RBF, reaches 93.45% cluster accuracy on UCF101, 27.79% on HMDB51, and 8.07% on SSv2, while four existing continual-learning methods adapted to the same unsupervised setting fall far behind. If correct, this shows that strong unsupervised video transformers can be turned into continual learners with almost no trainable parameters and without any task-specific bookkeeping.","feed_headline":"Label-free video learning hits 93% accuracy via density clusters","feed_subtitle":"No labels or task boundaries: kernel-density peaks on frozen video features keep old knowledge and add new clusters.","key_machinery":"Kernel Density Estimation (KDE) over deep-embedded video features: a non-parametric probability landscape where each data point places a Gaussian kernel. Mean-shift iteratively moves cluster centers to peaks of this landscape, discovering clusters without a pre-set count. Memory buffers (20 features per cluster) are replayed during later tasks, and a novelty threshold (Θ1 for feature distance, Θ2 for RBF softmax confidence) expands the cluster set. The RBF variant adds a linear layer trained with focal multi-class cross-entropy on pseudo-labels, which keeps trainable parameters near 100K while clusters grow.","core_discovery":"The central claim is that unsupervised video continual learning reduces to maintaining a non-parametric probability density over a frozen feature space. The method uses a video transformer to embed each clip, then applies mean-shift clustering over a Gaussian-kernel density estimate to discover cluster peaks without knowing the number of clusters. A small memory buffer per peak is replayed when the next task arrives, and a novelty detector—based on distance to existing centers (uVCL-KDE) or on softmax confidence of a linear readout (uVCL-KDE-RBF)—decides when to add a new cluster. The reported result is that the RBF variant substantially outperforms iCaRL, iCaRL+CL, EWC, and MAS on three act","pith_inferences":["The comparison is not apples-to-apples: the reported advantage may largely come from the much stronger frozen video transformer rather than from the KDE clustering itself; a k-means baseline on the same features would isolate the contribution.","The method's dependence on two hand-set thresholds (bandwidth h and novelty threshold Θ2) suggests an extension where those are adapted online—for example, from local density or replay confidence—could make it more robust across datasets.","Storing features rather than raw videos and replaying them into a KDE landscape transfers directly to image and audio continual learning, where the same non-parametric machinery applies.","If the frozen feature space already separates classes well, the method may be better described as unsupervised clustering with memory replay than as representation learning; its low accuracy on harder datasets like SSv2 hints at the ceiling imposed by the pretrained features."],"forward_implications":["Unsupervised video continual learning becomes a feature-space bookkeeping problem: no labels, task boundaries, or class counts are needed.","Because the feature extractor is frozen, the method inherits the quality of the pretrained video transformer; better unsupervised video representations should directly lift cluster accuracy.","The near-zero backward forgetting suggests that KDE replay stabilizes old clusters while new ones are added, addressing the stability-plasticity trade-off with a single memory buffer.","The mixed-category, 256-video-per-task protocol gives a reproducible benchmark for future unsupervised video continual learning work on UCF101, HMDB51, and SSv2.","With roughly 100K trainable parameters and training times under a day for 659 SSv2 tasks, the approach is practical for resource-constrained continual learning."],"supporting_citations":[{"why":"Supplies the frozen VideoMAE V2 transformer features that the clustering operates on.","marker":"[47]"},{"why":"Provides the kernel bandwidth estimation for the KDE representation.","marker":"[3]"},{"why":"Supplies the mean-shift clustering procedure used to find KDE peaks.","marker":"[8]"},{"why":"iCaRL is a principal baseline and the source of the exemplar-memory replay paradigm adapted here.","marker":"[38]"},{"why":"vCLIMB provides the video continual learning baseline and evaluation protocol adapted for unsupervised learning.","marker":"[45]"},{"why":"EWC is a regularization-based baseline compared against the proposed method.","marker":"[18]"},{"why":"MAS is a regularization-based baseline compared against the proposed method.","marker":"[1]"},{"why":"Focal loss is used to weight pseudo-clusters and handle imbalanced category sizes.","marker":"[25]"},{"why":"Radial basis function networks inspire the linear readout layer in uVCL-KDE-RBF.","marker":"[2]"}],"fun_headline_variants":["No labels, no boundaries: video model learns new tasks via density peaks","Kernel density clusters let video AI learn forever without forgetting","Unsupervised video learning: non-parametric clusters beat supervised baselines","Video continual learning without labels or task cues using KDE","Density-based clustering unlocks label-free video continual learning"],"cache_read_input_tokens":18432,"weakest_assumption_plain":"The load-bearing premise is that the reported accuracy gain comes from the KDE clustering and novelty detection, not from the fact that the proposed method uses a much stronger pretrained video transformer than the baselines; if a simple clustering method on the same features matches the results, the central claim does not follow.","fun_headline_variants_meta":{"raw":{"variants":["No labels, no boundaries: video model learns new tasks via density peaks","Kernel density clusters let video AI learn forever without forgetting","Unsupervised video learning: non-parametric clusters beat supervised baselines","Video continual learning without labels or task cues using KDE","Density-based clustering unlocks label-free video continual learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1543,"prompt_tokens":828,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":629}},"tokens_in":572,"tokens_out":715,"duration_ms":9545,"temperature":1.0,"reasoning_tokens":629,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:56:40.343407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the uVCL experiment on UCF101 with k-means or agglomerative clustering in place of mean-shift, using the same frozen video-transformer features, the same memory buffers of 20 features per cluster, the same 256-video mixed-class tasks, and the same RBF readout; if the cluster accuracy stays near 93%, then the KDE/mean-shift machinery is not the source of the improvement.","supporting_citations":[{"cited_title":"Videomae v2: Scaling video masked autoencoders with dual masking","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen VideoMAE V2 transformer features that the clustering operates on."},{"cited_title":"Bors and Nikolaos Nasios","cited_arxiv_id":null,"evidence_quote":"Provides the kernel bandwidth estimation for the KDE representation."},{"cited_title":"Mean shift: a robust approach toward feature space analysis","cited_arxiv_id":null,"evidence_quote":"Supplies the mean-shift clustering procedure used to find KDE peaks."},{"cited_title":"Lam- pert","cited_arxiv_id":null,"evidence_quote":"iCaRL is a principal baseline and the source of the exemplar-memory replay paradigm adapted here."},{"cited_title":"vCLIMB: A novel video class incremental learning benchmark","cited_arxiv_id":null,"evidence_quote":"vCLIMB provides the video continual learning baseline and evaluation protocol adapted for unsupervised learning."},{"cited_title":"Overcoming catastrophic forgetting in neural networks","cited_arxiv_id":null,"evidence_quote":"EWC is a regularization-based baseline compared against the proposed method."},{"cited_title":"Memory Aware Synapses: Learning what (not) to forget","cited_arxiv_id":null,"evidence_quote":"MAS is a regularization-based baseline compared against the proposed method."},{"cited_title":"Focal loss for dense object detection","cited_arxiv_id":null,"evidence_quote":"Focal loss is used to weight pseudo-clusters and handle imbalanced category sizes."},{"cited_title":"Minimal topology for a radial basis functions neural network for pattern classification","cited_arxiv_id":null,"evidence_quote":"Radial basis function networks inspire the linear readout layer in uVCL-KDE-RBF."}],"review_version":1}