{"id":"0bd7211d-c460-4026-a53f-d9ded91b444d","arxiv_id":"2412.04353","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A unified diffusion model with anticipative masking reaches state-of-the-art scores on action segmentation and anticipation, and the paper exposes a length-leakage flaw in prior anticipation benchmarks.","lead":"This paper proposes ActFusion, a single diffusion model that performs both temporal action segmentation and long-term action anticipation on the same video. It also reveals that standard anticipation benchmarks leaked the ground-truth video length during testing, and provides a corrected evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The anticipation-helps-segmentation claim rests on single-run ablations without variance estimates; the observed gaps (e.g., 88.5 vs. 87.2 in Table 4a) are comparable to typical seed noise and may not be statistically meaningful.","rationale":"The strongest claim is that a single unified model achieves state-of-the-art performance on both TAS and LTA and that the two tasks provide bi-directional benefits. The state-of-the-art numbers and the segmentation-helps-anticipation direction are reasonably supported by large margins and consistent trends. However, the anticipation-helps-segmentation direction is the load-bearing part of the 'bi-directional' claim, and the only quantitative support is Table 4a, which reports single-run ablations without any variance information. The gap of 1.3 points is within the range of typical seed variance for TAS on 50 Salads, and the paper provides no evidence that this gap exceeds run-to-run noise. The lack of released code also prevents the reader from cheaply checking the stability of the result. My recommendation is CONDITIONAL, matching the reader, with the condition being a multi-seed variance analysis or a softened claim. I do not see a more fundamental flaw: the method is described clearly, the diffusion formulation is standard, and the rectified LTA evaluation protocol is a genuine contribution. The concern is one of statistical support, not architectural soundness.","tokens_in":23306,"tokens_out":1408,"duration_ms":14032,"concrete_test":"Rerun the Table 4a ablation on 50 Salads, rows (1) and (2), with the same hyperparameters and data splits but at least 5 different random seeds each. Report mean and standard deviation of the TAS Avg metric (and F1@{10,25,50}) per condition. If the mean gap between the full-masking and no-M_A conditions is smaller than twice the pooled standard error, or if the full-model mean is within one standard deviation of the no-M_A mean, the paper should be revised to state that the improvement from anticipative masking on TAS is not statistically established. As a secondary check, verify that the reported 88.5 and 87.2 correspond to a single fixed seed and disclose which seed that was.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim of bi-directional benefit between TAS and LTA is supported primarily by ablation Table 4a: removing anticipative masking M_A drops the 50 Salads TAS average from 88.5 (row 1) to 87.2 (row 2), a 1.3-point gap. The reader correctly identifies this as the weakest load-bearing assumption: no standard deviations, no number of seeds, and no repeated-run information are reported anywhere in the paper for any ablation or main result. For action segmentation benchmarks, run-to-run variance from random initialization and training noise is commonly on the order of 1-2 F1 points, especially on small datasets like 50 Salads (50 videos, 5 splits). Given also the acknowledged text-table inconsistency in the Appendix (Table S4 and Table R4) and the absence of released code, the claim that anticipation helps segmentation is not distinguishable from training noise on the evidence presented. The LTA-side benefit of segmentation is much larger (e.g., row (1) vs row (3) in Table 3: 27.32 vs 39.55 at beta=0.1, alpha=0.2), so that direction is robust, but the reverse direction, which is essential to the 'bi-directional benefits' claim in the abstract and conclusion, is quantitatively fragile. This is not an argument that the claimed effect is false; it is an argument that the paper does not currently establish it with the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ActFusion, a unified diffusion model that jointly addresses temporal action segmentation (TAS) and long-term action anticipation (LTA). The key idea is to train an encoder--decoder with two masking strategies: anticipative masking, which hides a late part of the video and replaces it with learnable tokens to force future prediction, and random masking, which improves robustness against ambiguous visible frames. A single model is trained once with all five masking types and, at inference, uses no masking for TAS and anticipative masking for LTA. The authors claim state-of-the-art performance on 50 Salads, Breakfast, and GTEA for TAS, competitive or superior LTA results on 50 Salads and Breakfast, and bi-directional benefits between the two tasks. They also identify a protocol flaw in prior LTA evaluations that use ground-truth video length to set the prediction length, and they re-evaluate several methods under a rectified protocol without ground-truth length information.","tokens_in":23602,"tokens_out":6600,"duration_ms":61558,"significance":"If the claims are substantiated, this is a meaningful contribution: it demonstrates that a single unified model can outperform task-specific models on both TAS and LTA, and it highlights a previously underexamined source of information leakage in long-term action anticipation benchmarks. The paper provides a thorough set of ablations (mask types, losses, conditioning features, position embeddings, inference steps) and releases algorithmic details in the appendix, which strengthen the empirical study. However, the central bi-directional-benefit claim currently rests on a single-run ablation whose effect size is comparable to typical seed noise, and the rectified-protocol comparison is confounded by a training/inference asymmetry. These issues need to be resolved before the reported conclusions can be regarded as fully supported.","major_comments":[{"comment":"The conclusion that anticipation helps segmentation is based on the comparison of rows (1) and (2) in Table 4a: removing anticipative masking lowers the 50 Salads TAS average from 88.5 to 87.2, a 1.3-point gap. No standard deviations, number of seeds, or significance tests are reported anywhere in the paper, for the main results or for any ablation. On 50 Salads (50 videos, 5 splits), run-to-run variance in F1 scores from random initialization and training noise is commonly on the order of 1--2 points, so the observed gap may be indistinguishable from noise. Since the abstract and conclusion claim bi-directional benefits, this measurement is load-bearing. The authors should report mean and standard deviation over multiple random seeds for the main results and the key ablations, or provide a rigorous justification (e.g., paired significance testing) that the 1.3-point gap is not attributable to seed noise.","section":"Section 5.2, Table 4a"},{"comment":"The rectified-protocol comparison is asymmetric. ActFusion† is retrained with the modified anticipation masking strategy that uses prediction length rN_O, as stated in Section 5.4, while Cycle Cons.† and FUTR† are modified only at inference time to use the new prediction length while retaining their original training. The substantial performance drops for the baselines (e.g., FUTR† on 50 Salads, α=0.3, β=0.5, dropping from 15.26 to 7.48) may largely reflect that the models were not trained for the tested condition rather than an inherent inability to handle unknown prediction length. To support the claim that ActFusion outperforms baselines under the rectified protocol, the authors should retrain the baselines with a matching protocol (or, at minimum, report both inference-only and retrained baseline results) and explicitly disclose this asymmetry in the main text.","section":"Section 5.4, Table 6"},{"comment":"The training algorithm applies a signal scaling step, 'a_gt = (a_gt * 2 - 1) * scale', but no value for 'scale' is reported in the implementation details or the hyperparameter table. This parameter determines the range of the diffusion target and directly affects the loss scale and sampling behavior; without it, the reported numbers cannot be reproduced. Please specify the value of 'scale' for each dataset and, if it has a material effect, include an ablation over plausible values.","section":"Section B (Algorithm 1) and Table S5"}],"minor_comments":[{"comment":"The random-mask notation is inconsistent: the method defines a clip size Q and number of masked clips N_R, but Section 5 says 'we fix the size of patch w to 10' and Figure 3 mentions 'window size Q of each masked clip N_P to 10'; please harmonize the variable names and caption wording.","section":"Section 4.3 and Section 5 (Implementation details)"},{"comment":"The caption states that results are reported for observation ratio α=0.3, but the table header shows β(α=0.2); the caption and table should be aligned.","section":"Table 4b"},{"comment":"The text refers to 'Table R4' when presenting loss ablation results, but the corresponding table is labeled Table S3; please correct the cross-reference.","section":"Appendix E, 'Loss ablations'"},{"comment":"On 50 Salads with α=0.2 and β=0.1, ActFusion achieves 39.55, which ties with FUTR's 39.55; the abstract's wording 'outperforming task-specific models' is slightly too strong for this cell, although the overall average remains superior.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a well-defined method and extensive ablations, but the evidence for the bi-directional benefit is currently vulnerable on two fronts: missing variance estimates for a small effect and an unfair baseline comparison in the rectified-protocol experiment. Both issues are fixable within the scope of a revision. I would also encourage the authors to double-check the table and figure cross-references in the appendix before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the evaluation-protocol correction is the most valuable part: the authors show that several prior LTA methods leak ground-truth video length at test time by setting the prediction horizon to beta*T, and they propose a rectified protocol using r*N_O instead. That is a real service to the community and will likely affect how future LTA comparisons are run. Second, the central architectural claim—that one diffusion model can do both TAS and LTA through anticipative masking with learnable tokens—is plausible and mostly well supported, but the \"anticipation helps segmentation\" direction is quantitatively fragile.\n\nWhat is genuinely new: a single model, trained once, that achieves SOTA on both tasks across 50 Salads, Breakfast, and GTEA. The anticipative masking idea is simple and effective, and the ablation coverage is thorough: masking types, mask token position, conditioning features, position embeddings, loss ablations, inference steps. The paper also includes failure cases and an honest limitation statement about frame-wise accuracy being slightly below DiffAct. That is good empirical practice.\n\nThe soft spots are real but not disqualifying. No error bars or number of seeds are reported anywhere. The key ablation for the bi-directional benefit claim is Table 4a: removing anticipative masking drops 50 Salads TAS average from 88.5 to 87.2. On a 50-video dataset with 5 splits, that gap is within typical run-to-run noise from random initialization. The reverse direction—segmentation helping anticipation—is much larger (Table 3: 27.32 vs 39.55 at beta=0.1) and robust. So the abstract's \"bi-directional benefits\" claim is not established on the evidence as reported, though it may well be true. The rectified-protocol comparison (Table 6) is also not perfectly fair: ActFusion† is retrained with the modified masking, while FUTR† and Cycle Cons.† are only changed at inference. That said, the scale of the baselines' degradation strongly suggests the leakage was real.\n\nThere is one minor appendix error: the loss-ablation text refers to \"Table R4\" instead of the actual table number. Minor, but sloppy.\n\nWho this is for: anyone working on temporal action understanding, especially LTA evaluation. It deserves a serious referee. My recommendation: send it to review, but require the authors to (1) release code, (2) report multi-seed variance for the core ablations, and (3) either temper the bi-directional claim or back it with data. The protocol-correction contribution alone justifies a revised version.","headline":"A solid, useful unification of TAS and LTA with a genuinely valuable evaluation-protocol correction; the main weak spot is that the anticipation-helps-segmentation claim rests on single-run ablations without variance estimates.","tokens_in":24126,"tokens_out":1434,"would_cite":true,"duration_ms":16443,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single diffusion model segments observed actions and anticipates future actions in one pass.","keywords":["temporal action segmentation","long-term action anticipation","diffusion model","unified model","anticipative masking","joint learning","video understanding"],"falsifier":"Run the same ActFusion training on 50 Salads with several random seeds, with and without anticipative masking, and compare the spread of average TAS accuracy; if the difference between the two conditions is within the seed noise, the paper's core mutual-benefit claim fails. The same test applies to the LTA ablations, where removing anticipative masking drops the beta=0.5, alpha=0.2 score from 22.07 to 9.38.","tokens_in":23103,"feed_emoji":"🎬","tokens_out":4067,"duration_ms":38623,"temperature":0.7,"pith_summary":"Temporal action segmentation and long-term action anticipation are usually treated as separate problems with separate models. ActFusion claims that one diffusion-based model can do both at once: it classifies the visible frames of a video while predicting the invisible future frames, trained in a single process. The paper argues the two tasks help each other, so joint training improves both, and reports state-of-the-art results on standard benchmarks. If correct, this removes the need for task-specific architectures and makes a shared temporal understanding of actions feasible.","feed_headline":"One model masters action segmentation and anticipation together","feed_subtitle":"Joint training lifts both tasks past task-specific rivals on 50 Salads, Breakfast, and GTEA.","key_machinery":"Anticipative masking is the load-bearing mechanism: a binary mask hides a late part of the input frame sequence, and learnable mask tokens occupy the hidden positions so the encoder produces embeddings for both visible and invisible tokens, and the denoising decoder learns to reconstruct action labels for the whole sequence. Random masking, relation masking, and boundary masking are auxiliary strategies that make predictions robust to ambiguous or missing video parts. The architecture is an encoder-decoder transformer with dilated attention; the decoder follows the DDPM/DDIM diffusion framework, predicting clean action labels from noisy ones conditioned on the encoder embeddings.","core_discovery":"ActFusion's central claim is that action segmentation and long-term anticipation are two views of one generative task: reconstructing a complete action sequence from a partially observed one. The model is a diffusion decoder that iteratively denoises action-label sequences from Gaussian noise, conditioned on embeddings produced by an encoder fed with masked video features. During training, one of five masking strategies is randomly applied; the key one is anticipative masking, where a late portion of the sequence is replaced by learnable mask tokens so the model must infer the future. At inference, no masking is used for segmentation and anticipative masking for anticipation, so the identical weights handle both tasks. With this unified training, ActFusion reports state-of-the-art performance in both tasks on 50 Salads, Breakfast, and GTEA, and its ablations show segmentation loss improves anticipation and anticipative masking improves segmentation, i.e., bi-directional benefits. The paper also flags that some prior LTA evaluations use ground-truth video length at test time and demonstrates the method still leads under a rectified protocol.","pith_inferences":["If joint training works by learning temporal action structure, anticipative masking may serve as a general pretraining objective for other sequential action understanding tasks, such as early recognition or anomaly prediction, before fine-tuning.","The reported gains in anticipation from segmentation supervision suggest that stronger segmenters or boundary-supervised encoders could further improve long-horizon prediction; this is a testable hypothesis the paper does not run.","Since the paper's ablation numbers appear to lack variance estimates, a multi-seed replication is needed before the bi-directional benefit claim can be treated as quantitative rather than qualitative.","The rectified LTA evaluation suggests future benchmarks should ban ground-truth length and report prediction length as a function of observed frames only, which would change leaderboard comparisons."],"forward_implications":["A single trained model can be deployed for both tasks, eliminating the need for per-task architectures or per-task training.","Joint training gives bi-directional gains: segmentation supervision improves anticipation, and anticipative masking improves segmentation accuracy and segment metrics.","Masked training with learnable tokens lets the model keep performing when input parts are missing or ambiguous, not just for the two benchmark tasks.","LTA can be evaluated without ground-truth prediction length, and ActFusion retains its lead in that more realistic setting."],"supporting_citations":[{"why":"Supplies the diffusion action segmentation backbone and the relation and boundary masking strategies that ActFusion adapts.","marker":"[43]"},{"why":"Provides the transformer-based LTA baseline that ActFusion outperforms and uses as a cross-task comparison.","marker":"[25]"},{"why":"Represents the prior method that trains separate models for TAS and LTA, serving as the main unified-model baseline.","marker":"[52]"},{"why":"Defines the forward and reverse denoising diffusion process that ActFusion builds on.","marker":"[29]"},{"why":"Supplies the DDIM accelerated inference update rule used at test time.","marker":"[54]"},{"why":"Provides the standard TAS benchmark protocol, input features, and smoothing loss.","marker":"[19]"},{"why":"Provides the encoder-decoder transformer architecture that ActFusion modifies with dilated attention.","marker":"[65]"},{"why":"Defines the LTA evaluation protocol and benchmarks, including the ground-truth length leakage issue the paper rectifies.","marker":"[2]"},{"why":"Serves as an LTA baseline compared under the rectified evaluation without ground-truth prediction length.","marker":"[20]"}],"fun_headline_variants":["Unified diffusion model achieves both action segmentation and anticipation","Two tasks, one model: diffusion unifies segmentation and anticipation","Joint training lifts segmentation and anticipation past task-specific nets","ActFusion: one diffusion model, both tasks, SOTA on three benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim that anticipation helps segmentation rests on ablation gaps like 88.5 versus 87.2 in average TAS accuracy, reported without standard deviations; if run-to-run variance across random seeds is larger than that gap, the bi-directional benefit is not established.","fun_headline_variants_meta":{"raw":{"variants":["Unified diffusion model achieves both action segmentation and anticipation","Two tasks, one model: diffusion unifies segmentation and anticipation","Joint training lifts segmentation and anticipation past task-specific nets","ActFusion: one diffusion model, both tasks, SOTA on three benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2335,"prompt_tokens":939,"completion_tokens":1396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1325}},"tokens_in":555,"tokens_out":1396,"duration_ms":14181,"temperature":1.0,"reasoning_tokens":1325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:31:15.317062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ActFusion training on 50 Salads with several random seeds, with and without anticipative masking, and compare the spread of average TAS accuracy; if the difference between the two conditions is within the seed noise, the paper's core mutual-benefit claim fails. The same test applies to the LTA ablations, where removing anticipative masking drops the beta=0.5, alpha=0.2 score from 22.07 to 9.38.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the diffusion action segmentation backbone and the relation and boundary masking strategies that ActFusion adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the transformer-based LTA baseline that ActFusion outperforms and uses as a cross-task comparison."},{"cited_title":"Sener, D","cited_arxiv_id":null,"evidence_quote":"Represents the prior method that trains separate models for TAS and LTA, serving as the main unified-model baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the forward and reverse denoising diffusion process that ActFusion builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DDIM accelerated inference update rule used at test time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standard TAS benchmark protocol, input features, and smoothing loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the encoder-decoder transformer architecture that ActFusion modifies with dilated attention."},{"cited_title":"Abu Farha, A","cited_arxiv_id":null,"evidence_quote":"Defines the LTA evaluation protocol and benchmarks, including the ground-truth length leakage issue the paper rectifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as an LTA baseline compared under the rectified evaluation without ground-truth prediction length."}],"review_version":1}