{"id":"2f048a2b-d59f-4173-b35f-855dbe7a8dfb","arxiv_id":"2412.16946","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A replay-based knowledge distillation baseline outperforms most compared methods on three newly designed video domain incremental learning benchmarks for home action recognition.","lead":"The paper defines a video domain incremental learning benchmark built from three existing action recognition datasets, split by user, scene, or both, and proposes a replay plus knowledge distillation baseline. It shows this simple baseline beats most compared continual learning methods on the new splits, though some reported numbers in the buffer analysis contradict the paper's own tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark may not be a valid DIL benchmark because tasks may not share a fixed class set; the Toyota Smarthome class-count inconsistency signals this risk.","rationale":"The reader's verdict is CONDITIONAL and its weakest assumption is exactly the fixed-class-set protocol: each task Dk must contain all classes in C with sufficient samples. Our stress test identifies the same load-bearing concern: the benchmark's validity requires this protocol to hold, and the internal contradiction between Table 1 (55 classes) and Sec. 3.1 (31 activities) for Toyota Smarthome is concrete evidence that this assumption is not yet verified. Because the central claim is only meaningful if the benchmark is a valid DIL benchmark, this is the most load-bearing issue. The reader's conditional verdict appropriately captures the need for verification, so our verdict remains UNCHANGED; the concrete test we propose is precisely the check that would determine whether the paper can be upgraded to ACCEPT or should be rejected.","tokens_in":12989,"tokens_out":3346,"duration_ms":30709,"concrete_test":"For each dataset and split (NTU users, Toyota Smarthome scenes, ETRI hybrid), compute a class-coverage matrix: for every task k and every class c, count the number of samples of class c present in task k. Verify that every class appears in every task with a sufficient minimum (e.g., at least 1 sample, ideally >5 for stable evaluation). If any (task, class) entry is zero, the fixed-class-set assumption is violated. Additionally, reconcile Table 1: check the official Toyota Smarthome class count (31) and reproduce the benchmark's class list; if the benchmark uses 55 classes, document where they came from. This test would settle whether the proposed benchmark is a true DIL benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('our simple sampling and replay strategy outperforms most existing continual learning methods across the three proposed benchmarks') depends on the benchmark being a valid Domain-Incremental Learning (DIL) setup, where each task Dk contains all classes C as stated in the problem definition (Sec. 3). If tasks have incomplete or disjoint class sets, the setup degenerates into class-incremental or partial-domain learning, and the AA/BWF metrics no longer measure pure domain shift. The paper never verifies class coverage for any split, and there is direct evidence of unreliable benchmark construction: Table 1 lists 55 classes for Toyota Smarthome while Sec. 3.1 states the dataset has 31 activities. Moreover, Toyota Smarthome activities are naturally room-specific, so a 'scene domain' split will likely leave many classes absent from each scene, violating the fixed-class-set protocol. If the splits do not satisfy the protocol, DRIFT's comparative advantage could reflect class imbalance or label distribution shift rather than domain-incremental generalization, undermining the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes Video Domain Incremental Learning (VDIL) for human action recognition in home environments, a continual-learning setting in which the action-class set stays fixed while the data distribution shifts across tasks. It introduces three benchmarks built from public datasets by splitting NTU RGB+D into user domains, Toyota Smarthome into scene domains, and ETRI-Activity3D-LivingLab into hybrid user-scene domains. The authors propose DRIFT, a replay-and-distillation baseline that uses reservoir sampling and a dual classification-plus-KD loss, and compare it against Naive, Joint, LwF, EWC, DER, iCaRL, and BiC on average accuracy (AA) and backward forgetting (BWF). The reported results show that DRIFT outperforms all non-replay baselines and most replay baselines, with particularly large gains on Toyota Smarthome, and the paper includes ablations on loss components, sampling strategies, and memory size.","tokens_in":13155,"tokens_out":4824,"duration_ms":46281,"significance":"If the benchmark protocol is valid, this would be one of the first dedicated video domain-incremental learning benchmarks, addressing a real gap since most video continual-learning work targets class-incremental settings. The three split types are a sensible way to isolate user-driven, scene-driven, and combined domain shift, and the choice of widely used public action-recognition datasets makes the benchmark accessible. DRIFT is a simple and competitive task-agnostic baseline; on Toyota Smarthome it improves substantially over the compared methods, and its buffer-size analysis is a useful starting point for resource-constrained deployment. However, the central claim depends on the assumption that every task contains the full fixed class set, and the paper contains internal inconsistencies that call this assumption into question. The benchmark and the comparative results are potentially valuable, but the protocol must be verified and the reported numbers reconciled before the claims can be accepted.","major_comments":[{"comment":"The problem definition in Sec. 3 states that each task Dk encompasses all classes in C, but the paper never verifies this for any of the three splits, and Table 1 lists Toyota Smarthome as having 55 classes while Sec. 3.1 states the dataset has 31 activities. Because Toyota Smarthome activities are naturally room-specific, scene-domain splits may omit many classes from each task; if so, the AA and BWF metrics measure partial-domain or class-imbalanced learning rather than pure domain shift. Please report per-task class coverage and class-frequency statistics for all three benchmarks, and either restrict the evaluation to classes that appear in every task or explicitly justify why the fixed-class-set protocol is preserved.","section":"Sec. 3 and Table 1"},{"comment":"The buffer-size section states that a memory size of 550 achieves 83.17% and 'exceeds' Joint, but Table 3 shows that 83.17 is obtained at memory size 1100, while the 550 row reports 78.45. The same paragraph says 550 corresponds to 15.6% of the training set, which does not match 550/8,848 ≈ 6.2%. This inconsistency directly affects the paper's claim about matching or exceeding the non-incremental upper bound; please correct the table or the text and recompute the stated percentage.","section":"Sec. 4.3 and Table 3"},{"comment":"The main comparison in Table 2 does not specify the memory buffer size used for each method, and Sec. 4.1 only mentions method-specific hyperparameters. Without knowing whether all replay methods use the same buffer size and the same buffer-management rules, the headline claim that DRIFT outperforms most existing continual learning methods cannot be assessed fairly. Please state the buffer size used for DRIFT, iCaRL, DER, and BiC in the main experiments, and clarify how each method's buffer is populated.","section":"Sec. 4.1 and Table 2"},{"comment":"The knowledge-distillation loss in Eq. (5) is written as a cross-entropy between f_theta_{t-1}(V)/T and f_theta_t(V)/T, but dividing raw network outputs by T does not produce a valid probability distribution unless a softmax is applied to the logits before the temperature scaling. As written, the loss is not a standard distillation objective and its behavior is undefined for unbounded logits. Please clarify whether f denotes logits or probabilities, use softmax(z/T) consistently, and also explain why distillation is applied only to current-task samples and not to memory-buffer samples.","section":"Sec. 3.2, Eq. (5)"},{"comment":"The paper emphasizes that DRIFT is task-agnostic and 'does not require knowledge of task identifiers', but the evaluation protocol is not fully specified: the metrics ak,T in Eqs. (2)-(3) require evaluating each task separately, and it is unclear whether task identity is available at test time for any method. Please state explicitly whether test samples are evaluated with or without task IDs, and whether the iCaRL and BiC baselines are given task boundaries during training. This is essential for judging the claimed advantage of DRIFT in task-agnostic scenarios.","section":"Sec. 3.2 and Sec. 4.2"}],"minor_comments":[{"comment":"The word 'Hybird' in the last row of Table 1 should be 'Hybrid'.","section":"Table 1"},{"comment":"The table header contains the typo 'ackward Transfer' instead of 'Backward Transfer', and the text in Sec. 4.3 uses both BWF and BWT for the same metric; please standardize the terminology.","section":"Table 2"},{"comment":"The caption says the bar plots show catastrophic forgetting, but the figure appears to contain line plots or charts rather than bar plots; please align the caption with the actual figure.","section":"Fig. 1 caption"},{"comment":"The sentence describing the 550-memory result should be rewritten to match Table 3: the 83.17% value corresponds to memory size 1100, and the 550 value is 78.45%.","section":"Sec. 4.3"},{"comment":"Please state whether the TSN backbone is initialized with ImageNet-pretrained weights for all compared methods, and whether any method uses additional data augmentation besides the DER horizontal flips mentioned in the text, so that the comparison is fully reproducible.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark-protocol issue is the main risk. If the authors can verify fixed class coverage per task or re-run the experiments with a corrected protocol, the contribution is likely publishable; if not, the benchmark may reduce to a partial-domain or class-imbalance study. For a benchmark paper, the authors should also release the exact task splits, sample lists, and evaluation code, since none of these are currently provided. The internal contradictions in Table 1 and Sec. 3.1, and in Table 3 and Sec. 4.3, need to be resolved before the paper can be considered reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe benchmark idea is worth taking seriously, but the Toyota Smarthome scene split is probably not a valid domain-incremental setup, and the paper has a few internal inconsistencies that need fixing before the artifact can be trusted as a community resource.\n\nWhat's genuinely new: this is the first dedicated video domain-incremental learning benchmark for home environments, with user, scene, and hybrid splits built from three existing datasets. That fills a real gap for service robots and smart home systems. The DRIFT baseline - reservoir sampling replay plus knowledge distillation, no domain labels - is simple and appropriate as a default. The comparison covers the usual continual learning methods, and the main results are mostly internally consistent.\n\nThe soft spots are real but addressable. First, Table 1 lists 55 classes for Toyota Smarthome while Section 3.1 says 31 activities. That is not a trivial typo; it suggests benchmark construction was not carefully checked. Second, and more load-bearing, the scene split likely violates the fixed-class-set assumption. Toyota Smarthome activities are room-specific, so each camera/scene task probably contains only a subset of classes. The paper states in Section 3 that each task contains all classes C, but never verifies this for any split. If tasks have incomplete class sets, the AA and BWF metrics measure partial domain shifts, and the central claim about DRIFT's advantage is compromised. The stress-test note lands on this point, and I think it is correct.\n\nThird, the buffer-size section misattributes an AA of 83.17 to memory size 550 when Table 3 shows that result belongs to the 1100 setting, and the text says this exceeds Joint when Joint sits at 83.9. That is a concrete factual error. Fourth, there is no code, no exact splits, no seeds, and no error bars. For a benchmark paper, that is a serious omission.\n\nOne nuance: on NTU and ETRI, BiC beats DRIFT by about four points, so the headline claim is true only in the narrow sense of beating the other five baselines on most datasets. That is acceptable, but it should be stated precisely.\n\nWho is this for? Researchers working on continual learning for video, robotics, and smart home systems. The benchmark gap is real, and the paper deserves a serious referee. But it needs major revision: verify class coverage per split, fix the Toyota numbers, and release the splits and code.\n\nRecommendation: send to peer review.\n\nBest,","headline":"Useful VDIL benchmark idea, but the Toyota scene split likely breaks the fixed-class assumption and several internal inconsistencies need fixing before this is a reliable community resource.","tokens_in":13708,"tokens_out":4481,"would_cite":false,"duration_ms":36077,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper formalizes Video Domain Incremental Learning for home action recognition and shows that a simple replay baseline—reservoir sampling plus knowledge distillation, no domain labels—outperforms most existing continual learning…","keywords":["video domain incremental learning","action recognition","continual learning","catastrophic forgetting","replay buffer","reservoir sampling","home environments","domain shift"],"falsifier":"Compute, for each domain task in NTU RGB+D, Toyota Smarthome, and ETRI-Activity3D-LivingLab, the set of action classes actually present. If any task lacks a substantial class, the average-accuracy and backward-forgetting numbers conflate domain shift with class imbalance, and the benchmark does not measure what it claims. Resolving the 31-versus-55 class-count discrepancy for Toyota Smarthome would be a concrete first check.","tokens_in":12776,"feed_emoji":"🎥","tokens_out":4914,"duration_ms":40925,"temperature":0.7,"pith_summary":"Video Domain Incremental Learning (VDIL) is the problem of adapting an action-recognition model to a sequence of new domains—different users, rooms, or camera views—while the set of action classes stays fixed. This paper argues that home environments produce exactly such shifts, that they cause catastrophic forgetting, and that the problem has been overlooked relative to class-incremental video learning. To test this, the paper builds three benchmarks by splitting NTU RGB+D by user, Toyota Smarthome by scene, and ETRI-Activity3D-LivingLab by both. It then proposes DRIFT, a plain replay baseline that fills a memory buffer with reservoir sampling and trains with classification plus knowledge-distillation losses, requiring no domain labels. The central claim is that this simple strategy outperforms most existing continual learning methods across the three benchmarks, with the largest margin on the scene-based split.","feed_headline":"Simple replay beats most rivals on a new video domain-shift benchmark","feed_subtitle":"New benchmark splits home videos by user, scene, and both; a task-agnostic replay method wins most.","key_machinery":"DRIFT (Domain Replay Incremental Feature Training) is the central mechanism. Its key parts are reservoir sampling, which keeps the memory buffer a fair sample of all domains seen so far without needing to know where one domain ends and another begins, and a dual loss: a classification cross-entropy term over current and buffered samples, plus a knowledge-distillation term that pins the current model's softened outputs to the previous model's outputs on new data. The benchmark machinery consists of domain splits (user, scene, hybrid) over three RGB-D datasets with a fixed class set, evaluated by average accuracy and backward forgetting.","core_discovery":"In home action recognition, the dominant source of representation drift is the domain itself, not the action class: the paper's t-SNE visualization of Toyota Smarthome shows samples grouping by camera viewpoint rather than by activity. The paper therefore formalizes VDIL and constructs a benchmark where each task is one domain and every task shares the full class set, with average accuracy and backward forgetting as metrics. The proposed DRIFT baseline—reservoir sampling into a fixed memory plus cross-entropy on current and replayed data and knowledge distillation from the previous model—reaches 78.45% average accuracy on the scene-based split, ahead of all compared methods, and stays competitive on the user and hybrid splits while using no task identifiers. The accompanying ablations show the two losses are complementary and that random sample selection outperforms entropy-based and class-prototype selection for this setting.","pith_inferences":["The result suggests that for many real-world video deployments, sophisticated regularizers or task-boundary-aware buffers may be unnecessary; the bottleneck is storing representative old samples.","One could test the same benchmark protocol on egocentric or surveillance video, where domain shifts are driven by wearer or viewpoint rather than home scenes.","The benchmark's value depends on verifying that every split indeed contains all classes with enough samples; the paper does not show per-task class distributions, so the reported accuracy and forgetting numbers should be read with that caveat until the protocol is checked.","Future VDIL work could combine reservoir replay with prompt-based or parameter-efficient adapters, potentially closing the remaining gap to joint training while keeping the task-agnostic property."],"forward_implications":["A task-agnostic replay method with no domain labels is a viable default baseline for video domain-incremental learning, not a stopgap.","Scene-based domain shifts in home environments are harder than user-based shifts, so deployment systems should prioritize viewpoint and room diversity when collecting data.","Memory-constrained setups can use reservoir sampling: even a 225-sample buffer beats larger buffers of more complex methods on the scene benchmark.","The dual-loss recipe (cross-entropy on replay plus knowledge distillation) transfers directly to other video continual-learning settings with fixed classes."],"supporting_citations":[{"why":"Defines the three types of incremental learning and the domain-incremental setting this paper adopts.","marker":"[21]"},{"why":"NTU RGB+D is the dataset used to construct the user-based domain split.","marker":"[46]"},{"why":"Toyota Smarthome is the dataset used to construct the scene-based domain split.","marker":"[20]"},{"why":"ETRI-Activity3D-LivingLab is the dataset used to construct the hybrid domain split.","marker":"[47]"},{"why":"iCaRL is the replay baseline that DRIFT compares against and whose sample-management strategy is tested in the ablation.","marker":"[22]"},{"why":"LwF supplies the knowledge-distillation loss idea used in DRIFT's dual loss.","marker":"[27]"},{"why":"EWC is a regularization-based baseline that DRIFT must beat in the comparison.","marker":"[25]"},{"why":"BiC is the strongest competing baseline, which DRIFT outperforms on the scene benchmark while falling slightly short on the other two.","marker":"[28]"},{"why":"DER is a replay baseline that stores past logits and is part of the comparison set.","marker":"[23]"}],"fun_headline_variants":["Action recognition in homes: domain shifts, not classes, are the real challenge","Simple replay wins on new video domain-shift benchmark for home actions","How to keep home action recognition accurate as users and scenes change","New benchmark exposes domain shift as the true hurdle in home activity recognition","Task-agnostic replay with limited memory tops domain shift for home actions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes every task (domain) contains all action classes in sufficient number, so that the accuracy metrics measure pure domain shift; the paper states this assumption but never verifies it on the three datasets, and its own text gives contradictory class counts for Toyota Smarthome (31 activities in Section 3.1 versus 55 classes in Table 1).","fun_headline_variants_meta":{"raw":{"variants":["Action recognition in homes: domain shifts, not classes, are the real challenge","Simple replay wins on new video domain-shift benchmark for home actions","How to keep home action recognition accurate as users and scenes change","New benchmark exposes domain shift as the true hurdle in home activity recognition","Task-agnostic replay with limited memory tops domain shift for home actions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3385,"prompt_tokens":923,"completion_tokens":2462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2369}},"tokens_in":539,"tokens_out":2462,"duration_ms":16783,"temperature":1.0,"reasoning_tokens":2369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:57:07.156533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for each domain task in NTU RGB+D, Toyota Smarthome, and ETRI-Activity3D-LivingLab, the set of action classes actually present. If any task lacks a substantial class, the average-accuracy and backward-forgetting numbers conflate domain shift with class imbalance, and the benchmark does not measure what it claims. Resolving the 31-versus-55 class-count discrepancy for Toyota Smarthome would be a concrete first check.","supporting_citations":[{"cited_title":"Three types of incremental learning,","cited_arxiv_id":null,"evidence_quote":"Defines the three types of incremental learning and the domain-incremental setting this paper adopts."},{"cited_title":"NTU RGB+D: A large scale dataset for 3d human activity analysis,","cited_arxiv_id":null,"evidence_quote":"NTU RGB+D is the dataset used to construct the user-based domain split."},{"cited_title":"Toyota smarthome: Real-world activities of daily living,","cited_arxiv_id":null,"evidence_quote":"Toyota Smarthome is the dataset used to construct the scene-based domain split."},{"cited_title":"iCaRL: Incremental Classifier and Representation Learning,","cited_arxiv_id":null,"evidence_quote":"iCaRL is the replay baseline that DRIFT compares against and whose sample-management strategy is tested in the ablation."},{"cited_title":"Learning without Forgetting,","cited_arxiv_id":null,"evidence_quote":"LwF supplies the knowledge-distillation loss idea used in DRIFT's dual loss."},{"cited_title":"Overcoming catastrophic forgetting in neural net- works,","cited_arxiv_id":null,"evidence_quote":"EWC is a regularization-based baseline that DRIFT must beat in the comparison."},{"cited_title":"Large Scale Incremen- tal Learning,","cited_arxiv_id":null,"evidence_quote":"BiC is the strongest competing baseline, which DRIFT outperforms on the scene benchmark while falling slightly short on the other two."},{"cited_title":"Dark Experience for General Continual Learning: A Strong, Simple Baseline,","cited_arxiv_id":null,"evidence_quote":"DER is a replay baseline that stores past logits and is part of the comparison set."}],"review_version":1}