REVIEW 4 major objections 6 minor 25 references
Uncertainty-Aware Anticipation of Activities
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A video anticipation model samples many possible futures from learned action and length distributions.
desk verdict A solid incremental method for sampling-based long-term activity anticipation, but the multi-modality claim is not actually measured and the 'without compromising' line is contradicted by the clearly defined mode results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the factored autoregressive distribution over future action segments. The action model encodes the observed segments with two GRU layers and outputs a softmax distribution over the next action label; the length model encodes the same context plus a one-hot encoding of the sampled future action and outputs the mean and variance of a Gaussian duration $N(\mu, \sigma^2)$. The two are sampled alternately at test time, so the dependency between which action happens and how long it lasts is explicit, and uncertainty accumulates naturally as the prediction horizon grows.
What would settle it
Count the number of distinct predicted action sequences among the 25 samples on Breakfast and 50Salads, or measure coverage against a human-annotated set of plausible future segments; if most samples collapse to the same sequence while average accuracy stays high, the claim of multi-modal anticipation is not supported.
Extended reading notes
Core claim
The paper's central claim is that the probability of the next action segment factorizes as $p(c_{n+1}, l_{n+1} \mid c_{1:n}, l_{1:n}) = p(c_{n+1} \mid c_{1:n}, l_{1:n}) p(l_{n+1} \mid c_{1:n}, l_{1:n}, c_{n+1})$, with the first factor modeled by an RNN/GRU softmax classifier over action labels and the second by a Gaussian whose mean and variance are predicted conditioned on the sampled action. Feeding the sampled segment back lets the model roll forward to arbitrary horizons. The authors report that this sampling procedure outperforms n-gram/Gaussian baselines on both datasets, and that the mode of the distribution is comparable to dedicated single-sequence models while the top-1 accuracy over samples is higher.
Load-bearing premise
The claim that the model captures multi-modal futures rests on using average frame-wise accuracy over 25 samples as the evaluation metric, which does not measure whether the samples are distinct plausible alternatives.
Editorial extensions
If this is right
- The same trained model can produce any number of future sequences, and increasing the sample count mainly reduces the variance of the averaged score rather than changing the mean.
- The mode of the predicted distribution gives a deterministic single-sequence prediction, so uncertainty handling does not force a trade-off against standard anticipation accuracy.
- Because the length model is conditioned on the sampled action, the framework can predict not only what might happen but also for how long each alternative might last.
- The approach remains effective when the observed segment labels are inferred by an action recognition model instead of given as ground truth, which matters for real video input.
Reading between the lines
- The paper's evaluation would be strengthened by a coverage or diversity metric; average frame-wise accuracy over samples cannot distinguish a model that outputs many genuinely different futures from one that outputs near-duplicates.
- The same factorization could be applied to other structured sequence prediction problems with duration uncertainty, such as surgical step prediction or traffic maneuver anticipation.
- A planning system could use the sampled futures as a policy prior, optimizing over the distribution rather than a single predicted sequence.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an uncertainty-aware framework for long-term activity anticipation. It factorizes the predictive distribution over the next action segment into an action model (an RNN predicting a softmax over action labels) and a length model (an RNN predicting a Gaussian over segment duration, conditioned on the sampled future action). At test time the model recursively samples action-label/length pairs to generate multiple plausible future activity sequences, and a mode-based variant is used for single-sequence prediction. Experiments on Breakfast and 50Salads compare the sample-averaged mean-over-classes (MoC) accuracy against n-gram baselines, report mode accuracy against prior single-sequence methods, include a qualitative example, and analyze the effect of the number of samples. The paper claims that the model captures multi-modal futures without compromising single-sequence accuracy.
Significance. If the central claims are validated, the paper addresses an important limitation of prior long-term anticipation work: most existing methods output a single future sequence and thereby ignore the intrinsic uncertainty of long horizons. The autoregressive factorization in Eq. (1) is simple and reasonable, and the evaluation covers two standard datasets with both ground-truth and inferred observations. The consistent improvement over n-gram baselines on both datasets is a concrete positive result. However, the paper's headline claim, capturing multi-modal future activities, is not directly measured: the sample-averaged MoC used throughout Section 4 is a first-moment statistic that does not reward diversity, and the "Top-1 MoC" reported in Tables 6-7 is never defined. In addition, the mode-based single-sequence accuracy is consistently below prior methods, which contradicts the abstract's "without compromising" statement. These issues are load-bearing for the main claims, so the paper needs substantial revision before it can be accepted.
major comments (4)
- [Section 4, Evaluation Metric, and Tables 1-5] The evaluation protocol does not substantiate the multi-modality claim. The metric reported for sampled outputs is the mean-over-classes of the per-class frame-wise accuracy averaged over 25 generated samples. This is a first-moment quantity: a sampler that repeatedly emits the most probable sequence can achieve nearly the same score as a genuinely diverse sampler whenever the mode is accurate. The paper reports no measure of diversity or coverage, such as the number of distinct predicted sequences, pairwise sample diversity, oracle/top-k accuracy, or any distributional divergence between the predicted and ground-truth action distributions. Figure 3 is qualitative only. Without such a metric, the abstract's claim that the model "captures multi-modal future activities" is not supported by the experiments.
- [Tables 6-7 and Fig. 3 caption] The "Top-1 MoC" used to support the "without compromising accuracy when predicting a single sequence" claim is never defined in Section 4. The Fig. 3 caption states that samples are "ranked based on the frame-wise accuracy of the predicted activities," which suggests that Top-1 may be the best-of-N sample per video or per class. If so, it is an oracle metric that no fixed single-sequence policy can achieve, and it cannot be compared fairly with the deterministic RNN, CNN, and Time-Conditioned methods. The paper must define this metric precisely and either justify it as a comparison or remove it.
- [Abstract and Section 4.4, Tables 6-7] The claim that the model performs well "without compromising the accuracy when predicting a single sequence of future activities" is contradicted by the reported mode results. For example, on Breakfast with 20% observation and 10% prediction (ground-truth observations), Ours (Mode) is 0.5300 versus 0.6035 for the RNN model [1] and 0.6446 for Time-Cond. [8]; similar gaps appear throughout Tables 6 and 7. The paper's own discussion acknowledges this gap, stating that lower mode accuracy "is expected since these approaches were trained to predict only a single sequence." The abstract and conclusion should be revised to state that the mode accuracy is lower than the state of the art, and that the benefit lies specifically in multi-sample prediction.
- [Section 4.4 and Table 8] The comparison with the variational auto-encoder method [15] on next-segment prediction reports only label accuracy (57.8 versus 62.2). Since [15] predicts a distribution over action and starting time, a comparison on label accuracy alone does not isolate the multi-modality aspects of either method. If the paper claims to model uncertainty better than [15], a diversity or distributional comparison for this short-term setup would be needed; otherwise the discussion of this comparison should be tempered.
minor comments (6)
- [Section 1, Introduction] The phrase "However , as increasing the predicted time horizon" contains a typo and a grammar issue; it should read "However, as the predicted time horizon increases."
- [Eq. (11), Section 3.3] The notation in Eq. (11) is confusing: the normalized length is written as l = (l - l)/σ_l, using the same symbol l for both the raw and normalized length. A different symbol, such as l', would improve readability.
- [Section 4, Evaluation Metric] The sentence "To evaluate multiple samples of future activities, the average frame-wise accuracy of each action class is used to compute the MoC" should clarify whether the average is computed over the 25 generated sequences for each test video and then aggregated over videos, or pooled globally. The current wording is ambiguous.
- [Section 4.3, Table 5] The reported mean and standard deviation over 5 runs for different sample counts would benefit from a statistical significance test or confidence intervals, especially on 50Salads where the standard deviation is relatively large.
- [Fig. 3] The figure shows 25 samples, but only a subset is visually discriminable; consider highlighting the distinct modes separately or providing a compact representation of distinct sampled sequences.
- [Related Work, Section 2] The discussion of Furnari et al. [3], which evaluates top-k accuracy for multi-modal future anticipation, is relevant to the paper's methodology but is not connected to the choice of evaluation metrics in Section 4. A brief discussion of why the paper does not use top-k accuracy would be helpful.
Circularity Check
No circularity: the anticipation model is trained and evaluated on external datasets against published baselines; self-citations are component reuse, not load-bearing proof.
full rationale
The paper's derivation chain is self-contained: the action model p(c_{n+1}|c_{1:n}, l_{1:n}) and length model N(\mu, \sigma^2) are trained with cross-entropy and negative log-likelihood losses on ground-truth segmentations, then evaluated on held-out splits of the Breakfast and 50Salads datasets against n-gram baselines and published methods ([1], [8], [15]). The authors' prior work [1] is used as a building block for the observation-inference step and the RNN architecture, but it is not invoked as a proof of the new uncertainty-sampling result; the novel contribution is the action/length distribution modeling and recursive sampling, whose reported numbers come from test-time evaluation. The paper does not fit a parameter to a subset and then rename it a prediction, nor does any equation reduce to its own input by construction. A measurement-validity concern remains: the sample-averaged MoC metric does not by itself certify diversity, and 'Top-1 MoC' is never defined, so the multi-modality claim is only qualitatively supported. That is a weakness in evaluation, not a circular derivation, and no load-bearing claim rests solely on a self-citation chain.
Assumptions & free parameters
assumptions (4)
- standard math Probability factorization: p(c_n+1, l_n+1 | ...) = p(c_n+1 | ...) * p(l_n+1 | ..., c_n+1)
- domain assumption Action lengths follow a Gaussian distribution: p(l | ...) = N(mu, sigma^2)
- domain assumption The two-step paradigm of [1]: observed actions are first inferred by an RNN-HMM, then future actions are predicted
- domain assumption The observed action segments and lengths are sufficient for predicting the future (autoregressive condition)
Cite this review
Pith. "Pith review of Uncertainty-Aware Anticipation of Activities." pith.science (2026). https://pith.science/paper/E244NKYH
@misc{pith2026190809540,
author = {Pith},
title = {Pith review of: Uncertainty-Aware Anticipation of Activities},
year = {2026},
howpublished = {\url{https://pith.science/paper/E244NKYH}},
note = {Machine review of arXiv:1908.09540}
}
read the original abstract
Anticipating future activities in video is a task with many practical applications. While earlier approaches are limited to just a few seconds in the future, the prediction time horizon has just recently been extended to several minutes in the future. However, as increasing the predicted time horizon, the future becomes more uncertain and models that generate a single prediction fail at capturing the different possible future activities. In this paper, we address the uncertainty modelling for predicting long-term future activities. Both an action model and a length model are trained to model the probability distribution of the future activities. At test time, we sample from the predicted distributions multiple samples that correspond to the different possible sequences of future activities. Our model is evaluated on two challenging datasets and shows a good performance in capturing the multi-modal future activities without compromising the accuracy when predicting a single sequence of future activities.
Figures
Reference graph
Works this paper leans on
-
[1]
When will you do what?-Anticipating temporal occurrences of activities
Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what?-Anticipating temporal occurrences of activities. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5343–5352, 2018. 1, 2, 3, 4, 5, 6, 7
work page 2018
-
[8]
Time- conditioned action anticipation in one shot
Qiuhong Ke, Mario Fritz, and Bernt Schiele. Time- conditioned action anticipation in one shot. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,
-
[15]
A variational auto- encoder model for stochastic point processes
Nazanin Mehrasa, Akash Abdu Jyothi, Thibaut Durand, Ji- awei He, Leonid Sigal, and Greg Mori. A variational auto- encoder model for stochastic point processes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2, 5, 6, 7
work page 2019
-
[2]
Bayesian prediction of future street scenes using synthetic likelihoods
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Bayesian prediction of future street scenes using synthetic likelihoods. In International Conference on Learning Repre- sentations (ICLR), 2019. 5
work page 2019
-
[3]
Antonino Furnari, Sebastiano Battiato, and Giovanni Maria Farinella. Leveraging uncertainty to rethink loss functions and evaluation measures for egocentric action anticipation. In European Conference on Computer Vision Workshops , pages 389–405. Springer, 2018. 2
work page 2018
-
[4]
RED: Rein- forced encoder-decoder networks for action anticipation
Jiyang Gao, Zhenheng Yang, and Ram Nevatia. RED: Rein- forced encoder-decoder networks for action anticipation. In British Machine Vision Conference (BMVC), 2017. 1, 2
work page 2017
-
[5]
Action completion: A temporal model for moment detection
Farnoosh Heidarivincheh, Majid Mirmehdi, and Dima Damen. Action completion: A temporal model for moment detection. In British Machine Vision Conference (BMVC) ,
-
[6]
Max-margin early event detectors
Minh Hoai and Fernando De la Torre. Max-margin early event detectors. International Journal of Computer Vision , 107(2):191–202, 2014. 1
work page 2014
Show all 25 references
-
[7]
Zamir, Silvio Savarese, and Ashutosh Saxena
Ashesh Jain, Amir R. Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-RNN: Deep learning on spatio-temporal graphs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1
2016
-
[9]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 4
2015
-
[10]
Anticipating hu- man activities using object affordances for reactive robotic response
Hema S Koppula and Ashutosh Saxena. Anticipating hu- man activities using object affordances for reactive robotic response. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 38(1):14–29, 2016. 1
2016
-
[11]
The language of actions: Recovering the syntax and semantics of goal- directed human activities
Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal- directed human activities. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) , pages 780– 787, 2014. 1, 4
2014
-
[12]
A hier- archical representation for future action prediction
Tian Lan, Tsung-Chuan Chen, and Silvio Savarese. A hier- archical representation for future action prediction. In Euro- pean Conference on Computer Vision (ECCV) , pages 689–
-
[13]
Learning ac- tivity progression in LSTMs for activity detection and early detection
Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning ac- tivity progression in LSTMs for activity detection and early detection. In IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 1942–1950, 2016. 1
1942
-
[14]
Joint prediction of activity labels and starting times in untrimmed videos
Tahmida Mahmud, Mahmudul Hasan, and Amit K Roy- Chowdhury. Joint prediction of activity labels and starting times in untrimmed videos. In IEEE International Confer- ence on Computer Vision (ICCV) , pages 5773–5782, 2017. 2
2017
-
[16]
Leveraging the present to anticipate the future in videos
Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran. Leveraging the present to anticipate the future in videos. In IEEE Conference on Com- puter Vision and Pattern Recognition Workshops, 2019. 2
2019
-
[17]
Automatic dif- ferentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic dif- ferentiation in pytorch. In Advances in Neural Information Processing Systems Workshops, 2017. 4
2017
-
[18]
Weakly supervised action learning with RNN based fine-to-coarse modeling
Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with RNN based fine-to-coarse modeling. In IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2017. 2, 5
2017
-
[19]
Ac- tion anticipation by predicting future dynamic images
Cristian Rodriguez, Basura Fernando, and Hongdong Li. Ac- tion anticipation by predicting future dynamic images. In European Conference on Computer Vision Workshops, pages 89–105. Springer, 2018. 2
2018
-
[20]
Human activity prediction: Early recog- nition of ongoing activities from streaming videos
Michael S Ryoo. Human activity prediction: Early recog- nition of ongoing activities from streaming videos. In IEEE International Conference on Computer Vision (ICCV), pages 1036–1043, 2011. 1
2011
-
[21]
Encouraging LSTMs to anticipate actions very early
Mohammad Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, and Lars Andersson. Encouraging LSTMs to anticipate actions very early. In IEEE International Conference on Computer Vision (ICCV), 2017. 1
2017
-
[22]
Action an- ticipation with RBF kernelized feature mapping RNN
Yuge Shi, Basura Fernando, and Richard Hartley. Action an- ticipation with RBF kernelized feature mapping RNN. In European Conference on Computer Vision (ECCV) , pages 301–317, 2018. 2
2018
-
[23]
Combining em- bedded accelerometers with computer vision for recogniz- ing food preparation activities
Sebastian Stein and Stephen J McKenna. Combining em- bedded accelerometers with computer vision for recogniz- ing food preparation activities. In ACM International Joint Conference on Pervasive and Ubiquitous Computing , pages 729–738, 2013. 1, 4
2013
-
[24]
An- ticipating visual representations from unlabeled video
Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. An- ticipating visual representations from unlabeled video. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 98–106, 2016. 1
2016
-
[25]
Visual forecasting by imitating dy- namics in natural sequences
Kuo-Hao Zeng, William B Shen, De-An Huang, Min Sun, and Juan Carlos Niebles. Visual forecasting by imitating dy- namics in natural sequences. In IEEE International Confer- ence on Computer Vision (ICCV) , pages 2999–3008, 2017. 2
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.