Pith. sign in

REVIEW 3 major objections 6 minor 52 references

Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraints

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TCCA adds a bi-directional context regularizer and a learned-transition CRF to a parallel query-based decoder, and the paper reports state-of-the-art or comparable long-term action anticipation on four benchmarks.

desk verdict A solid empirical extension of FUTR with two plausible consistency modules, but the missing query-to-segment matching description leaves the central mechanism unverified; deserves a revision, not rejection. read the letter →

arxiv 2412.19424 v1 pith:AY26FFCP submitted 2024-12-27 cs.CV

Alberto Maté , Mariella Dimiccoli This is my paper · ORCID
classification cs.CV
keywords long-termactionanticipationtemporalconsistencyconditionalrandomfieldtransitionmatrixsegmentationtransformerencoder-decoderparalleldecodingvideounderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that long-term action anticipation—predicting the labels and durations of future actions from an observed video prefix—is improved most by enforcing temporal context consistency, not just by better features or bigger decoders. It introduces TCCA, an encoder-decoder transformer that combines a specialized action-segmentation encoder for the observed past with a parallel query-based decoder, and adds two modules: a bi-directional regularizer that supervises each predicted segment against its neighbors, and a learned transition matrix used in a conditional random field that optimizes the whole predicted sequence globally. On four benchmarks, Breakfast, 50Salads, EpicKitchens-55, and EGTEA+, the paper reports state-of-the-art or comparable results, including against probabilistic and large-language-model methods that take trimmed video as input. If correct, the improvement over the FUTR-style baseline shows that temporal-consistency constraints carry the gain.

What carries the argument

The load-bearing machinery is the pairing of the Bi-Directional Action Context Regularizer (BACR) with a CRF layer over the decoder queries. BACR adds two heads to each query, one predicting the previous action and one predicting the next action, and trains them with a divergence loss against the neighboring queries' present-action logits; the first query's previous head is tied to the last segmentation logits from the encoder. The CRF defines a score for an entire action sequence as the sum of per-query emission logits plus a term from a learned transition matrix, is trained by negative log-likelihood, and is decoded at inference with a dynamic-programming search. The argument depends on the K decoder queries being in chronological order, with queries after an end-of-sequence token discarded, so that adjacency in query index matches adjacency in time.

What would settle it

Inspect the training code to see how each query is matched to a ground-truth action segment; if matching is positional, then permuting the queries before computing the BACR and CRF losses should destroy the reported gains, whereas if the model still performs as well under such a permutation, the claimed temporal-consistency mechanism is not what carries the improvement.

Watch

Extended reading notes

Core claim

The central claim is that TCCA, by learning and enforcing temporal constraints, predicts future action sequences better than the underlying parallel decoder alone. The encoder uses a sparse-and-windowed attention segmentation model with a temporal smoothing loss to produce cleaner observations of the past; the decoder's queries are supervised in both directions, with each query predicting the previous and next action so that neighboring queries agree, and a CRF layer with a learned transition matrix scores whole sequences rather than independent actions. The paper reports that each module contributes: ablations show both regularizer directions together outperform either alone, and removing the CRF hurts every prediction horizon tested. The conclusion is that local and global temporal consistency, not any single component, is what carries the method to its reported results.

Load-bearing premise

The temporal-consistency modules only make sense if the decoder's K queries are ordered chronologically, so that each query's 'next' label really is the following query's 'present' label and the CRF transition matrix scores the sequence in that order; the paper does not explain how queries are matched to ground-truth segments during training.

Editorial extensions

If this is right

  • If the reported gains hold, parallel query-based decoders for anticipation should include explicit adjacency supervision and global sequence scoring as a default rather than an add-on.
  • Cleaner action segmentation of the observed past becomes a direct lever for anticipation accuracy, since the paper shows segmentation losses and encoder choice change anticipation results.
  • Learned transition matrices can substitute for hand-built temporal-logic rules, because the CRF discovers valid action transitions from data rather than requiring curated constraints.
  • The results on EpicKitchens-55 and EGTEA+ suggest temporal consistency helps even in multi-label settings where the evaluation metric does not reward ordering.
  • Because the CRF scores full sequences, label and duration predictions become jointly constrained, which matters for planning-oriented applications of anticipation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the learned transition matrix may encode activity-specific procedural knowledge; a testable extension is whether a matrix trained on one kitchen activity transfers to another, which would reveal whether the constraints are general or task-specific.
  • Editorial inference: the paper does not specify how query slots are matched to ground-truth action segments during training; if matching is not order-preserving, the BACR adjacency terms and the CRF transition scores would supervise arbitrary pairs rather than true temporal neighbors.
  • Editorial inference: a stress test with deliberately permuted queries at inference would separate the contribution of the transition matrix from the contribution of query ordering, showing which component actually enforces temporal consistency.
  • Editorial inference: the supplementary duration analysis shows 50Salads duration predictions collapse toward a narrow range; enforcing duration-transition consistency, such as expected durations per action class, is a natural next place for temporal constraints to help.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes TCCA, a transformer encoder-decoder for long-term action anticipation from untrimmed videos, building on the FUTR-style parallel query decoder. The two main contributions are a Bi-Directional Action Context Regularizer (BACR) that enforces local temporal consistency between adjacent predicted action segments via KL-divergence losses, and a CRF layer with a learned transition matrix that globally optimizes the predicted action sequence. The architecture also replaces the standard encoder with an LTContext action segmentation encoder. The method is evaluated on Breakfast, 50Salads, EpicKitchens-55, and EGTEA+, reporting MoC and mAP, with component ablations in Tables 3-10.

Significance. If the query-ordering concerns are resolved, this is a solid empirical contribution. The paper provides the first application of a segment-level CRF to long-term action anticipation and a clear ablation of the temporal-consistency modules, including comparisons against diffusion-based and LLM-based methods. The ablations are reasonably thorough, spanning four datasets and both segmentation and anticipation metrics, and the planned code release would aid reproducibility. The main weakness is that the core temporal-consistency mechanisms are not fully specified: the BACR and CRF losses are defined over query indices, and the manuscript never states how queries are matched to ground-truth segments, leaving the central claim vulnerable.

major comments (3)
  1. [Section 3.1, Eqs. (7)-(9)] The BACR and CRF modules are defined over the index order of the K decoder queries: Eq. (7) matches query i's future head to query i+1's present head, Eq. (8) matches query i's past head to query i-1's present head, and Eq. (9) scores a CRF over a_1...a_N. The manuscript never states how the decoder queries are matched to ground-truth action segments during training. Because the decoder is described as query-based and DETR-style query decoders are permutation-invariant and typically require bipartite matching, it is not evident that query i corresponds to the i-th future action in chronological order. If a non-chronological assignment is used, the KL losses in Eqs. (7)-(8) compare non-adjacent actions and the CRF in Eq. (9) decodes over an arbitrary permutation, so the improvements in Tables 5-9 could not be attributed to temporal-consistency constraints. Please specify the assignment rule explicitly; if queries are assigned in chronological order, state that clearly and describe how EOS-padded positions are masked in the losses.
  2. [Section 3.1, Eq. (8)] The first term of Eq. (8) is written as KL(a^{i=1}_{past} || F^{αT}_{seg}), but F_seg is defined earlier as a feature tensor of shape αT × D_seg, not as a vector of class logits. The surrounding text says the first query's previous head is supervised using 'the last logits of the action segmentation encoder', but the notation does not reflect this, and the superscript αT is not defined as a time-step selection. Please clarify the target of this term and define the indexing (e.g., the last frame's logits from the final encoder stage).
  3. [Section 3.1, Eq. (9) and inference] The CRF score in Eq. (9) includes the sum Σ_{i=0}^{N} M_{a_i,a_{i+1}}, which implicitly references a_0 and a_{N+1}, but no start/end tokens are defined. At inference, the number of future actions N is unknown; the paper does not state whether Viterbi decoding runs over the full K query positions and then truncates at the first <EOS> token, or whether a fixed maximum length is used, or how the transition matrix treats the <EOS> and start tokens. These details are necessary to reproduce the method and to interpret Table 7, since the CRF is claimed to optimize the global sequence.
minor comments (6)
  1. [Section 1, contributions bullet] The bullet claiming that 'the proposed method sets new state-of-the-art on four standard benchmarks' is not supported by Table 1: on Breakfast with α=0.2, GTAN (deterministic) achieves higher MoC than TCCA at β=0.2, 0.3, and 0.5, and at α=0.3, β=0.5 GTAN also wins. Please soften the claim to 'state-of-the-art or comparable performance on four benchmarks', as the abstract does.
  2. [Table 1] Several entries in Table 1 are typeset without separators between values (e.g., '39.5527.54', '22.5022.50', '29.6022.50'), making the numbers hard to parse. Please fix the formatting so each cell contains a single numeric value.
  3. [Eq. (4)] The duration loss Ldur is averaged over all K queries, but for queries beyond the <EOS> token there is no ground-truth duration. Please specify how the target durations for padded positions are set (e.g., zero) or how the loss is masked.
  4. [Table 9b] In the 50Salads row for the DETR encoder, the reported Edit score is 3.99 and F1@50 is 2.99, which are dramatically lower than for the other encoders and seem implausibly low; please verify these numbers or add a note explaining the failure mode of the DETR encoder on this dataset.
  5. [Section 4.3 (BACR ablation)] The section header reads 'BARC Loss' in the main text; this should be 'BACR Loss'. The same typo appears in the discussion of Table 6.
  6. [Section 3.1, Eq. (8)] The notation 'a^{i=1}_{past}' in Eq. (8) should be 'a^{1}_{past}' for consistency with the rest of the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TCCA's novel losses and learned transition matrix are standard supervised objectives, not predictions fitted to their own outputs.

full rationale

The paper's derivation chain is self-contained supervised training. BACR (Eqs. 5-8) supervises auxiliary 'previous/next' heads against the present-head logits of adjacent queries, which are themselves supervised through the CRF loss (Eqs. 9-10); this is a regularizer, not a definition of the target. The CRF transition matrix M is learned from ground-truth action sequences via negative log-likelihood, and Table 8 explicitly compares random versus precomputed initialization, showing the learned matrix is not forced by the initialization; the ablation in Table 7 shows the CRF component's contribution empirically rather than assuming it. The encoder choice (LTContext) is borrowed from external work [4] and ablated in Table 4 against DETR and FACT, so the segmentation-to-anticipation claim is supported by comparison, not by construction. No load-bearing self-citation appears: references [4], [10], [18], and [46] are external to the present authors, and no uniqueness claim is imported from prior work by the same authors. The reviewer-flagged ambiguity about how decoder queries are matched to ground-truth segments is a reproducibility/correctness concern about an unspecified training detail, not a case where a prediction reduces to its own fitted input. Overall, the reported improvements are empirical outcomes of supervised training with held-out test splits, so no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are standard hyperparameters and the learned transition matrix is a model parameter fitted to training data. The main load-bearing assumptions are the temporal ordering of queries, the sufficiency of I3D features, the Markov property of the CRF, and the continuity of the last observed action into the first predicted segment.

free parameters (4)
  • omega (CRF transition weight) = 0.1 for Breakfast, 1 for other datasets
    Hand-chosen per dataset; controls the strength of the transition matrix in Eq. (9). Not swept or justified beyond the reported values.
  • lambda (temporal smoothness weight) = 0.2 for all datasets
    Fixed hyperparameter for the smoothing loss Ls in Eq. (12). Chosen without a sensitivity analysis.
  • Number of anticipation queries K = 8 (Breakfast), 20 (50Salads), 200 (EpicKitchens-55), 30 (EGTEA)
    Set based on the number of actions to predict per dataset. Affects maximum sequence length and CRF complexity. The appendix tests 6/8/10 for Breakfast.
  • Sample rate and batch size = Sample rates 3/2/1 and batch sizes 16/8/4/8 for Breakfast/50Salads/EpicKitchens/EGTEA
    Chosen per dataset for training stability; reported in Appendix 9 without justification.
assumptions (4)
  • domain assumption Query indices correspond to chronological action segments
    The BACR loss (Eqs. 7-8) and CRF (Eq. 9) assume query i represents the i-th future segment in temporal order. The paper does not describe how this order is enforced during training (Section 3.1, Decoder and BACR paragraphs).
  • domain assumption I3D features from [10,39] are sufficient input representations
    The method uses pre-extracted I3D features for all datasets (Section 4.1, Training). No fine-tuning of the visual backbone is performed, so the claim depends on these features capturing the needed visual information.
  • domain assumption Action sequences are first-order Markov for the CRF
    The transition matrix M in Eq. (9) models only adjacent action transitions, independent of position and visual input. This is a strong simplification that the ablations support empirically but do not derive.
  • domain assumption The last observed action continues into the first future segment in most cases
    The first term of Eq. (8) supervises the first query's past head with the last segmentation logits from the observation. This assumes continuity across the observation/prediction boundary, which the authors acknowledge as typical but not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraints." pith.science (2026). https://pith.science/paper/AY26FFCP

@misc{pith2026241219424,
  author       = {Pith},
  title        = {Pith review of: Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AY26FFCP}},
  note         = {Machine review of arXiv:2412.19424}
}
read the original abstract

This paper proposes a method for long-term action anticipation (LTA), the task of predicting action labels and their duration in a video given the observation of an initial untrimmed video interval. We build on an encoder-decoder architecture with parallel decoding and make two key contributions. First, we introduce a bi-directional action context regularizer module on the top of the decoder that ensures temporal context coherence in temporally adjacent segments. Second, we learn from classified segments a transition matrix that models the probability of transitioning from one action to another and the sequence is optimized globally over the full prediction interval. In addition, we use a specialized encoder for the task of action segmentation to increase the quality of the predictions in the observation interval at inference time, leading to a better understanding of the past. We validate our methods on four benchmark datasets for LTA, the EpicKitchen-55, EGTEA+, 50Salads and Breakfast demonstrating superior or comparable performance to state-of-the-art methods, including probabilistic models and also those based on Large Language Models, that assume trimmed video as input. The code will be released upon acceptance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 48 canonical work pages

  1. [1]

    Uncertainty-aware anticipation of activities

    Yazan Abu Farha and Juergen Gall. Uncertainty-aware anticipation of activities. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision Workshops, pages 0–0, 2019

    work page 2019

  2. [2]

    Long-term antic- ipation of activities with cycle consistency

    Yazan Abu Farha, Qiuhong Ke, Bernt Schiele, and Juergen Gall. Long-term antic- ipation of activities with cycle consistency. In Pattern Recognition: 42nd DAGM German Conference, DAGM GCPR 2020, T¨ ubingen, Germany, September 28–October 1, 2020, Proceedings 42, pages 159–173. Springer, 2021

    work page 2020

  3. [3]

    When will you do what?- anticipating temporal occurrences of activ- ities

    Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what?- anticipating temporal occurrences of activ- ities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5343–5352, 2018

    work page 2018

  4. [4]

    How much temporal long-term context is needed for action segmentation? In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 10351–10361, 2023

    Emad Bahrami, Gianpiero Francesca, and Juergen Gall. How much temporal long-term context is needed for action segmentation? In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 10351–10361, 2023

    work page 2023

  5. [5]

    Unified fully and timestamp supervised tem- poral action segmentation via sequence to sequence translation

    Nadine Behrmann, S Alireza Golestaneh, Zico Kolter, J¨ urgen Gall, and Mehdi Noroozi. Unified fully and timestamp supervised tem- poral action segmentation via sequence to sequence translation. In Proceedings of the European Conference on Computer Vision, pages 52–68. Springer, 2022

    work page 2022

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    work page 1901

  7. [7]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kir- illov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Pro- ceedings of the European Conference on Com- puter Vision, pages 213–229. Springer, 2020

    work page 2020

  8. [8]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017

    work page 2017

Show all 52 references
  1. [9]

    Temporal action segmentation: An analysis of modern techniques

    Guodong Ding, Fadime Sener, and Angela Yao. Temporal action segmentation: An analysis of modern techniques. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2023

    2023

  2. [10]

    Ms-tcn: Multi-stage temporal convolutional network for action segmentation

    Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3575–3584, 2019

    2019

  3. [11]

    G.D. Forney. The viterbi algorithm. Proceed- ings of the IEEE, 61(3):268–278, 1973

    1973

  4. [12]

    Leveraging tem- poral context in low representational power regimes

    Camilo L Fosco, SouYoung Jin, Emilie Josephs, and Aude Oliva. Leveraging tem- poral context in low representational power regimes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10693–10703, 2023

    2023

  5. [13]

    Forecasting future action sequences with neural mem- ory networks

    Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Forecasting future action sequences with neural mem- ory networks. In Proceedings of the British Machine Vision Conference, 2019

    2019

  6. [14]

    Antic- ipative video transformer

    Rohit Girdhar and Kristen Grauman. Antic- ipative video transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13505–13515, Octo- ber 2021

    2021

  7. [15]

    Actionvlad: Learning spatio-temporal aggre- gation for action classification

    Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell. Actionvlad: Learning spatio-temporal aggre- gation for action classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 971– 980, 2017

    2017

  8. [16]

    Actfusion: a unified diffusion model for action segmentation and anticipation

    Dayoung Gong, Suha Kwak, and Minsu Cho. Actfusion: a unified diffusion model for action segmentation and anticipation. Advances in Neural Information Processing Systems, 2024. 11

    2024

  9. [17]

    Activity grammars for temporal action segmentation

    Dayoung Gong, Joonseok Lee, Deunsol Jung, Suha Kwak, and Minsu Cho. Activity grammars for temporal action segmentation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [18]

    Future transformer for long-term action anticipation

    Dayoung Gong, Joonseok Lee, Manjin Kim, Seong Jong Ha, and Minsu Cho. Future transformer for long-term action anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 3052–3061, 2022

    2022

  11. [19]

    A-act: Action anticipation through cycle transformations

    Akash Gupta, Jingen Liu, Liefeng Bo, Amit K Roy-Chowdhury, and Tao Mei. A-act: Action anticipation through cycle transformations. arXiv preprint arXiv:2204.00942, 2022

    2022 arXiv

  12. [20]

    Gaussian error linear units (gelus), 2023

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023

    2023

  13. [21]

    Bidi- rectional lstm-crf models for sequence tag- ging, 2015

    Zhiheng Huang, Wei Xu, and Kai Yu. Bidi- rectional lstm-crf models for sequence tag- ging, 2015

    2015

  14. [22]

    Timeception for complex action recognition

    Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. Timeception for complex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 254– 263, 2019

    2019

  15. [23]

    Videograph: Rec- ognizing minutes-long human activities in videos

    Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. Videograph: Rec- ognizing minutes-long human activities in videos. In ICCV Workshop, 2019

    2019

  16. [24]

    Time-Conditioned Action Anticipation in One Shot

    Qiuhong Ke, Mario Fritz, and Bernt Schiele. Time-Conditioned Action Anticipation in One Shot. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019

    2019

  17. [25]

    Palm: Predicting actions through language models, 2024

    Sanghwan Kim, Daoji Huang, Yongqin Xian, Otmar Hilliges, Luc Van Gool, and Xi Wang. Palm: Predicting actions through language models, 2024

    2024

  18. [26]

    The language of actions: Recovering the syn- tax and semantics of goal-directed human activities

    Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syn- tax and semantics of goal-directed human activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 780–787, 2014

    2014

  19. [27]

    An end-to-end generative framework for video segmentation and recognition

    Hilde Kuehne, Juergen Gall, and Thomas Serre. An end-to-end generative framework for video segmentation and recognition. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 1–8, 2016

    2016

  20. [28]

    A hybrid rnn-hmm approach for weakly supervised temporal action segmenta- tion

    Hilde Kuehne, Alexander Richard, and Juer- gen Gall. A hybrid rnn-hmm approach for weakly supervised temporal action segmenta- tion. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 42(4):765–779, 2018

    2018

  21. [29]

    On information and sufficiency

    Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951

    1951

  22. [30]

    Conditional random fields: Probabilistic models for segmenting and labeling sequence data

    John Lafferty, Andrew McCallum, Fernando Pereira, et al. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Icml, volume 1, page 3. Williamstown, MA, 2001

    2001

  23. [31]

    Tem- poral convolutional networks for action seg- mentation and detection

    Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Tem- poral convolutional networks for action seg- mentation and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 156– 165, 2017

    2017

  24. [32]

    Ms-tcn++: Multi-stage temporal convolutional network for action segmentation

    Shijie Li, Yazan Abu Farha, Yun Liu, Ming- Ming Cheng, and Juergen Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 45(6):6647–6658, 2020

    2020

  25. [33]

    In the eye of beholder: Joint learning of gaze and actions in first person video

    Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European Conference on Computer Vision, pages 619–635, 2018

    2018

  26. [34]

    SGDR: stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In Proceedings of the Interna- tional Conference on Learning Representa- tions, 2017

    2017

  27. [35]

    Decou- pled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decou- pled weight decay regularization. In Inter- national Conference on Learning Representa- tions, 2019

    2019

  28. [36]

    Fact: Frame- action cross-attention temporal modeling for efficient action segmentation

    Zijia Lu and Ehsan Elhamifar. Fact: Frame- action cross-attention temporal modeling for efficient action segmentation. In Proceed- ings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 18175–18185, June 2024

    2024

  29. [37]

    End- to-end fine-grained action segmentation and recognition using conditional random field 12 models and discriminative sparse coding

    Effrosyni Mavroudi, Divya Bhaskara, Shahin Sefati, Haider Ali, and Ren´ e Vidal. End- to-end fine-grained action segmentation and recognition using conditional random field 12 models and discriminative sparse coding. In Proceedings of the IEEE/CVF Winter Con- ference on Applic...

    2018

  30. [38]

    Can’t make an omelette without breaking some eggs: Plau- sible action anticipation using large video- language models

    Himangi Mittal, Nakul Agarwal, Shao-Yuan Lo, and Kwonjoon Lee. Can’t make an omelette without breaking some eggs: Plau- sible action anticipation using large video- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 185...

    2024

  31. [39]

    Ego- topo: Environment affordances from egocen- tric video

    Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego- topo: Environment affordances from egocen- tric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 163–172, 2020

    2020

  32. [40]

    Rethinking learning approaches for long-term action anticipation

    Megha Nawhal, Akash Abdu Jyothi, and Greg Mori. Rethinking learning approaches for long-term action anticipation. In Pro- ceedings of the European Conference on Com- puter Vision, pages 558–576. Springer, 2022

    2022

  33. [41]

    Temporal aggregate representations for long-range video understanding

    Fadime Sener, Dipika Singhania, and Angela Yao. Temporal aggregate representations for long-range video understanding. In Proceed- ings of the European Conference on Com- puter Vision, pages 154–171. Springer, 2020

    2020

  34. [42]

    Combining embedded accelerometers with computer vision for recognizing food prepa- ration activities

    Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food prepa- ration activities. In Proceedings of the 2013 ACM International Joint Conference on Per- vasive and Ubiquitous Computing, pages 729– 738, 2013

    2013

  35. [43]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017

    2017

  36. [44]

    Vamos: Versatile action models for video understanding

    Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for video understanding. In Proceedings of the Euro- pean Conference on Computer Vision, 2023

    2023

  37. [45]

    Don’t pour cereal into coffee: Differentiable temporal logic for temporal action segmen- tation

    Ziwei Xu, Yogesh Rawat, Yongkang Wong, Mohan S Kankanhalli, and Mubarak Shah. Don’t pour cereal into coffee: Differentiable temporal logic for temporal action segmen- tation. Advances in Neural Information Processing Systems, 35:14890–14903, 2022

    2022

  38. [46]

    Gated temporal diffusion for stochastic long-term dense anticipation

    Olga Zatsarynna, Emad Bahrami, Yazan Abu Farha, Gianpiero Francesca, and Juergen Gall. Gated temporal diffusion for stochastic long-term dense anticipation. In Proceedings of the European Conference on Computer Vision, 2024

    2024

  39. [47]

    Object-centric video represen- tation for long-term action anticipation

    Ce Zhang, Changcheng Fu, Shijie Wang, Nakul Agarwal, Kwonjoon Lee, Chiho Choi, and Chen Sun. Object-centric video represen- tation for long-term action anticipation. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 6751–6761, 2024

    2024

  40. [48]

    On diverse asynchronous activity anticipation

    He Zhao and Richard P Wildes. On diverse asynchronous activity anticipation. In Pro- ceedings of the European Conference on Com- puter Vision, pages 781–799. Springer, 2020

    2020

  41. [49]

    Antgpt: Can large language models help long-term action antic- ipation from videos?, 2024

    Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwon- joon Lee, and Chen Sun. Antgpt: Can large language models help long-term action antic- ipation from videos?, 2024

    2024

  42. [50]

    A sur- vey on deep learning techniques for action anticipation, 2023

    Zeyun Zhong, Manuel Martin, Michael Voit, Juergen Gall, and J¨ urgen Beyerer. A sur- vey on deep learning techniques for action anticipation, 2023

    2023

  43. [51]

    Diffant: Diffusion mod- els for action anticipation

    Zeyun Zhong, Chengzhi Wu, Manuel Martin, Michael Voit, Juergen Gall, and J¨ urgen Beyerer. Diffant: Diffusion mod- els for action anticipation. arXiv preprint arXiv:2311.15991, 2023. 13 7 Additional Ablation Study We have extended the ablation study presented in Section 4.3 to...

    2023 arXiv

  44. [52]

    after using their official checkpoints. The left side of the diagram displays action segmentation from the observation, while the right side shows action anticipation after decoding the action and duration into a frame-wise sequence. Results from different datasets, activities...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.