{"id":"d1fb2716-06e6-4f4e-b76c-8f7862a61b9a","arxiv_id":"1908.09540","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces a sampling-based framework with an action model and a length model to anticipate multiple future activity sequences and their durations.","lead":"This paper trains a neural network to predict a range of possible future activities in a video, and samples many alternatives at test time. It matters for robots and assistants that plan ahead and need to know the set of plausible futures, not just a single guess.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-modality claim rests on sample-averaged MoC, which does not reward diversity; the undefined 'Top-1 MoC' cannot support the 'without compromising' claim.","rationale":"The reader's weakest assumption identifies exactly the load-bearing weakness: sample-averaged MoC cannot certify multi-modality, because it does not measure diversity or coverage. My reading agrees and adds that the only place the paper claims to match single-sequence state of the art is an undefined 'Top-1 MoC' that appears to be oracle-selected from the samples. Both issues strike at the two halves of the abstract's central claim: (1) capturing multi-modal futures and (2) doing so without compromising single-sequence accuracy. The proposed concrete test would settle the first issue with a direct diversity/coverage comparison against a mode-only control; the second is settled by defining and recomputing Top-1 MoC under a fixed policy. A conditional verdict is the appropriate disposition: the method is clearly described, reproducible in principle, and outperforms n-gram baselines, but the headline claims require a corrected evaluation before acceptance. I do not see grounds for rejection, because a diversity metric could reveal genuine multi-modality, and the mode-based numbers, while below the single-sequence SOTA, are not catastrophic. Thus the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":10597,"tokens_out":3319,"duration_ms":35074,"concrete_test":"Re-run the evaluation on the existing 25-sample outputs using a diversity-sensitive protocol: for each test video, count the number of distinct full action-label sequences among the samples, and compute pairwise edit distance between sampled sequences and per-class recall of ground-truth future segments. Compare these against a degenerate control that draws 25 samples by repeatedly choosing the mode action and mean length (with identical runtime). If the control's sample-averaged MoC is comparable to the model's while its diversity is near zero, the multi-modality claim is unsupported. Additionally, inspect the code or redefine 'Top-1 MoC' in Section 4; if it is an oracle/best-of-samples selection, recompute Tables 6 and 7 using the defined mode as the single-sequence policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that sampling from the learned action and length distributions captures 'multi-modal future activities'—is not actually measured. Section 4 reports MoC as frame-wise accuracy averaged over classes and, for sampled outputs, averaged over 25 generated samples (Tables 1, 3, 5). This is a first-moment statistic: a sampler that repeatedly emits the mode (or any single high-probability sequence) can obtain nearly the same sample-averaged MoC as a genuinely diverse sampler whenever the mode is accurate. The metric is insensitive to whether samples cover distinct futures such as pour water versus pour coffee, which is precisely the phenomenon the paper motivates in Fig. 1. No coverage, number of distinct predicted sequences, pairwise diversity, or oracle/top-k accuracy is reported; Fig. 3 is qualitative only. Thus the multi-modality claim lacks quantitative support under the stated evaluation protocol.\n\nA second, entangled issue is the paper's 'without compromising accuracy when predicting a single sequence' assertion. In Tables 6 and 7, the evidence is 'Ours (Top-1) MoC', but this metric is never defined in Section 4. The Fig. 3 caption says samples are 'ranked based on the frame-wise accuracy of the predicted activities', suggesting Top-1 may be the best-of-N samples per video or per class—an oracle metric that no fixed single-sequence policy can achieve. The clearly defined Mode results in Tables 2 and 4 are consistently lower than the RNN model [1] and Time-Cond. [8] in Tables 6 and 7 (e.g., Breakfast, 20% observation, 50% prediction: Mode 0.3490 versus RNN 0.4042). So the only explicitly defined single-sequence numbers contradict the abstract's 'without compromising', while the numbers that support it are undefined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an uncertainty-aware framework for long-term activity anticipation. It factorizes the predictive distribution over the next action segment into an action model (an RNN predicting a softmax over action labels) and a length model (an RNN predicting a Gaussian over segment duration, conditioned on the sampled future action). At test time the model recursively samples action-label/length pairs to generate multiple plausible future activity sequences, and a mode-based variant is used for single-sequence prediction. Experiments on Breakfast and 50Salads compare the sample-averaged mean-over-classes (MoC) accuracy against n-gram baselines, report mode accuracy against prior single-sequence methods, include a qualitative example, and analyze the effect of the number of samples. The paper claims that the model captures multi-modal futures without compromising single-sequence accuracy.","tokens_in":10854,"tokens_out":2812,"duration_ms":30408,"significance":"If the central claims are validated, the paper addresses an important limitation of prior long-term anticipation work: most existing methods output a single future sequence and thereby ignore the intrinsic uncertainty of long horizons. The autoregressive factorization in Eq. (1) is simple and reasonable, and the evaluation covers two standard datasets with both ground-truth and inferred observations. The consistent improvement over n-gram baselines on both datasets is a concrete positive result. However, the paper's headline claim, capturing multi-modal future activities, is not directly measured: the sample-averaged MoC used throughout Section 4 is a first-moment statistic that does not reward diversity, and the \"Top-1 MoC\" reported in Tables 6-7 is never defined. In addition, the mode-based single-sequence accuracy is consistently below prior methods, which contradicts the abstract's \"without compromising\" statement. These issues are load-bearing for the main claims, so the paper needs substantial revision before it can be accepted.","major_comments":[{"comment":"The evaluation protocol does not substantiate the multi-modality claim. The metric reported for sampled outputs is the mean-over-classes of the per-class frame-wise accuracy averaged over 25 generated samples. This is a first-moment quantity: a sampler that repeatedly emits the most probable sequence can achieve nearly the same score as a genuinely diverse sampler whenever the mode is accurate. The paper reports no measure of diversity or coverage, such as the number of distinct predicted sequences, pairwise sample diversity, oracle/top-k accuracy, or any distributional divergence between the predicted and ground-truth action distributions. Figure 3 is qualitative only. Without such a metric, the abstract's claim that the model \"captures multi-modal future activities\" is not supported by the experiments.","section":"Section 4, Evaluation Metric, and Tables 1-5"},{"comment":"The \"Top-1 MoC\" used to support the \"without compromising accuracy when predicting a single sequence\" claim is never defined in Section 4. The Fig. 3 caption states that samples are \"ranked based on the frame-wise accuracy of the predicted activities,\" which suggests that Top-1 may be the best-of-N sample per video or per class. If so, it is an oracle metric that no fixed single-sequence policy can achieve, and it cannot be compared fairly with the deterministic RNN, CNN, and Time-Conditioned methods. The paper must define this metric precisely and either justify it as a comparison or remove it.","section":"Tables 6-7 and Fig. 3 caption"},{"comment":"The claim that the model performs well \"without compromising the accuracy when predicting a single sequence of future activities\" is contradicted by the reported mode results. For example, on Breakfast with 20% observation and 10% prediction (ground-truth observations), Ours (Mode) is 0.5300 versus 0.6035 for the RNN model [1] and 0.6446 for Time-Cond. [8]; similar gaps appear throughout Tables 6 and 7. The paper's own discussion acknowledges this gap, stating that lower mode accuracy \"is expected since these approaches were trained to predict only a single sequence.\" The abstract and conclusion should be revised to state that the mode accuracy is lower than the state of the art, and that the benefit lies specifically in multi-sample prediction.","section":"Abstract and Section 4.4, Tables 6-7"},{"comment":"The comparison with the variational auto-encoder method [15] on next-segment prediction reports only label accuracy (57.8 versus 62.2). Since [15] predicts a distribution over action and starting time, a comparison on label accuracy alone does not isolate the multi-modality aspects of either method. If the paper claims to model uncertainty better than [15], a diversity or distributional comparison for this short-term setup would be needed; otherwise the discussion of this comparison should be tempered.","section":"Section 4.4 and Table 8"}],"minor_comments":[{"comment":"The phrase \"However , as increasing the predicted time horizon\" contains a typo and a grammar issue; it should read \"However, as the predicted time horizon increases.\"","section":"Section 1, Introduction"},{"comment":"The notation in Eq. (11) is confusing: the normalized length is written as l = (l - l)/σ_l, using the same symbol l for both the raw and normalized length. A different symbol, such as l', would improve readability.","section":"Eq. (11), Section 3.3"},{"comment":"The sentence \"To evaluate multiple samples of future activities, the average frame-wise accuracy of each action class is used to compute the MoC\" should clarify whether the average is computed over the 25 generated sequences for each test video and then aggregated over videos, or pooled globally. The current wording is ambiguous.","section":"Section 4, Evaluation Metric"},{"comment":"The reported mean and standard deviation over 5 runs for different sample counts would benefit from a statistical significance test or confidence intervals, especially on 50Salads where the standard deviation is relatively large.","section":"Section 4.3, Table 5"},{"comment":"The figure shows 25 samples, but only a subset is visually discriminable; consider highlighting the distinct modes separately or providing a compact representation of distinct sampled sequences.","section":"Fig. 3"},{"comment":"The discussion of Furnari et al. [3], which evaluates top-k accuracy for multi-modal future anticipation, is relevant to the paper's methodology but is not connected to the choice of evaluation metrics in Section 4. A brief discussion of why the paper does not use top-k accuracy would be helpful.","section":"Related Work, Section 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a genuinely useful incremental paper, but the headline claim about multi-modal futures is not supported by the evaluation, and the abstract oversells the single-sequence accuracy. If you read it as 'we sample multiple futures from learned action/length distributions and it beats n-gram baselines,' it holds up. If you read it as 'we capture diverse plausible futures without hurting accuracy,' the evidence is thin.\n\nWhat's new and good: the idea is straightforward—train an action model and a Gaussian length model, feed them recursively, sample at test time to get multiple predicted future sequences. That direct sampling-based formulation is not in the cited prior work, which mostly outputs a single sequence. The experiments are careful in one important way: they compare against n-gram baselines with the same sampling protocol, and the proposed model wins consistently on both Breakfast and 50Salads, with and without ground-truth observations. The paper is clearly written, the method is reproducible from the text, and the qualitative example in Fig. 3 does show samples that look plausibly diverse.\n\nSoft spots, in proportion: first, the multi-modality claim is not measured. The reported MoC is averaged over 25 samples per class—a first-moment statistic. A sampler that always emits the mode can score nearly as well as a genuinely diverse sampler whenever the mode is accurate. No coverage, distinct-sequence count, pairwise diversity, or oracle top-k metric appears. So the central motivation in Fig. 1 (pour coffee vs. pour water vs. pour milk) is never quantitatively validated.\n\nSecond, the 'without compromising' assertion is contradicted by the clearly defined numbers. The Mode results in Tables 2 and 4 are consistently below the RNN model [1] and Time-Cond. [8] in Tables 6 and 7. The numbers that would support the abstract's claim are 'Ours (Top-1) MoC,' which is never defined in Section 4. The Fig. 3 caption says samples are ranked by frame-wise accuracy, which strongly suggests Top-1 is best-of-N oracleselection—not a fixed single-sequence policy. That is a real evaluation gap, not a nitpick.\n\nThird, minor: the comparison with [15] in Table 8 is dismissed with 'our approach is designed for long-term prediction,' but that is not a measured advantage; it is just a lower short-term number with a hand-wave.\n\nWho this is for: researchers working on long-term video anticipation or stochastic sequence prediction in activity recognition will want to know this method and its baselines. It is a reasonable building block, and the negative result on mode-vs-sampling tension is worth reporting. It deserves a serious referee, but the revision must define Top-1 MoC, add a diversity/coverage metric, and soften the abstract to match what is actually shown.\n\nRecommendation: send it to review, with the expectation of major revision on the evaluation protocol. The core sampling idea is sound; the evidence for the central claim is not.\n\nBest,\n[You]","headline":"A solid incremental method for sampling-based long-term activity anticipation, but the multi-modality claim is not actually measured and the 'without compromising' line is contradicted by the clearly defined mode results.","tokens_in":11415,"tokens_out":858,"would_cite":true,"duration_ms":9848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video anticipation model samples many possible futures from learned action and length distributions.","keywords":["action anticipation","long-term prediction","uncertainty modelling","multi-modal future","activity segmentation","Gaussian length model","recurrent neural networks","video understanding"],"falsifier":"Count the number of distinct predicted action sequences among the 25 samples on Breakfast and 50Salads, or measure coverage against a human-annotated set of plausible future segments; if most samples collapse to the same sequence while average accuracy stays high, the claim of multi-modal anticipation is not supported.","tokens_in":10387,"feed_emoji":"🎬","tokens_out":5730,"duration_ms":55063,"temperature":0.7,"pith_summary":"The paper argues that long-term video anticipation should be formulated as distribution learning rather than single-output prediction, because several minutes ahead the future is genuinely uncertain. It trains an action model and a length model that factorize the probability of the next action segment, then samples from those distributions recursively to produce many plausible future activity sequences. On the Breakfast and 50Salads datasets, the sampled sequences match several possible continuations, and the mode of the distribution still works as a single-sequence predictor. The payoff, if the claim holds, is a single framework that can both enumerate alternative futures for planning and give a best-guess sequence when one is required.","feed_headline":"Sampling predicts many plausible futures, not just one","feed_subtitle":"An action model plus a Gaussian length model generates diverse future sequences while keeping single-path accuracy.","key_machinery":"The central object is the factored autoregressive distribution over future action segments. The action model encodes the observed segments with two GRU layers and outputs a softmax distribution over the next action label; the length model encodes the same context plus a one-hot encoding of the sampled future action and outputs the mean and variance of a Gaussian duration $N(\\mu, \\sigma^2)$. The two are sampled alternately at test time, so the dependency between which action happens and how long it lasts is explicit, and uncertainty accumulates naturally as the prediction horizon grows.","core_discovery":"The paper's central claim is that the probability of the next action segment factorizes as $p(c_{n+1}, l_{n+1} \\mid c_{1:n}, l_{1:n}) = p(c_{n+1} \\mid c_{1:n}, l_{1:n}) p(l_{n+1} \\mid c_{1:n}, l_{1:n}, c_{n+1})$, with the first factor modeled by an RNN/GRU softmax classifier over action labels and the second by a Gaussian whose mean and variance are predicted conditioned on the sampled action. Feeding the sampled segment back lets the model roll forward to arbitrary horizons. The authors report that this sampling procedure outperforms n-gram/Gaussian baselines on both datasets, and that the mode of the distribution is comparable to dedicated single-sequence models while the top-1 accuracy over samples is higher.","pith_inferences":["The paper's evaluation would be strengthened by a coverage or diversity metric; average frame-wise accuracy over samples cannot distinguish a model that outputs many genuinely different futures from one that outputs near-duplicates.","The same factorization could be applied to other structured sequence prediction problems with duration uncertainty, such as surgical step prediction or traffic maneuver anticipation.","A planning system could use the sampled futures as a policy prior, optimizing over the distribution rather than a single predicted sequence."],"forward_implications":["The same trained model can produce any number of future sequences, and increasing the sample count mainly reduces the variance of the averaged score rather than changing the mean.","The mode of the predicted distribution gives a deterministic single-sequence prediction, so uncertainty handling does not force a trade-off against standard anticipation accuracy.","Because the length model is conditioned on the sampled action, the framework can predict not only what might happen but also for how long each alternative might last.","The approach remains effective when the observed segment labels are inferred by an action recognition model instead of given as ground truth, which matters for real video input."],"supporting_citations":[{"why":"Supplies the two-step anticipation framework, the RNN/CNN single-sequence baselines, and the observation/prediction percentage evaluation protocol used throughout the paper.","marker":"[1]"},{"why":"Provides the RNN-HMM model used to infer observed action segments when ground truth is not available.","marker":"[18]"},{"why":"Provides the Breakfast dataset used for evaluation.","marker":"[11]"},{"why":"Provides the 50Salads dataset used for evaluation.","marker":"[23]"},{"why":"Defines the next-action-segment accuracy comparison and the variational auto-encoder baseline for predicting a distribution over future actions.","marker":"[15]"},{"why":"Provides the time-conditioned single-sequence state-of-the-art method against which the mode of the predicted distribution is compared.","marker":"[8]"},{"why":"Justifies averaging frame-wise accuracy over samples as an evaluation practice in future prediction tasks.","marker":"[2]"}],"fun_headline_variants":["Sample many futures, not just one","Long-term video: sampling captures multiple possible futures","Diverse futures from video: action+length model with sampling","Uncertainty in video prediction? Sample to see all paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the model captures multi-modal futures rests on using average frame-wise accuracy over 25 samples as the evaluation metric, which does not measure whether the samples are distinct plausible alternatives.","fun_headline_variants_meta":{"raw":{"variants":["Sample many futures, not just one","Long-term video: sampling captures multiple possible futures","Diverse futures from video: action+length model with sampling","Uncertainty in video prediction? Sample to see all paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1221,"prompt_tokens":853,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":469,"tokens_out":368,"duration_ms":4295,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:07:49.701845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the number of distinct predicted action sequences among the 25 samples on Breakfast and 50Salads, or measure coverage against a human-annotated set of plausible future segments; if most samples collapse to the same sequence while average accuracy stays high, the claim of multi-modal anticipation is not supported.","supporting_citations":[{"cited_title":"When will you do what?-Anticipating temporal occurrences of activities","cited_arxiv_id":null,"evidence_quote":"Supplies the two-step anticipation framework, the RNN/CNN single-sequence baselines, and the observation/prediction percentage evaluation protocol used throughout the paper."},{"cited_title":"Weakly supervised action learning with RNN based fine-to-coarse modeling","cited_arxiv_id":null,"evidence_quote":"Provides the RNN-HMM model used to infer observed action segments when ground truth is not available."},{"cited_title":"The language of actions: Recovering the syntax and semantics of goal- directed human activities","cited_arxiv_id":null,"evidence_quote":"Provides the Breakfast dataset used for evaluation."},{"cited_title":"Combining em- bedded accelerometers with computer vision for recogniz- ing food preparation activities","cited_arxiv_id":null,"evidence_quote":"Provides the 50Salads dataset used for evaluation."},{"cited_title":"A variational auto- encoder model for stochastic point processes","cited_arxiv_id":null,"evidence_quote":"Defines the next-action-segment accuracy comparison and the variational auto-encoder baseline for predicting a distribution over future actions."},{"cited_title":"Time- conditioned action anticipation in one shot","cited_arxiv_id":null,"evidence_quote":"Provides the time-conditioned single-sequence state-of-the-art method against which the mode of the predicted distribution is compared."},{"cited_title":"Bayesian prediction of future street scenes using synthetic likelihoods","cited_arxiv_id":null,"evidence_quote":"Justifies averaging frame-wise accuracy over samples as an evaluation practice in future prediction tasks."}],"review_version":1}