Pith. sign in

REVIEW 1 major objections 4 minor 14 references

An Evaluation of Action Recognition Models on EPIC-Kitchens

T0 review · 1 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This report benchmarks TSN, TRN, and TSM on EPIC-Kitchens and finds that temporal-reasoning models outperform frame-level aggregation, while action accuracy remains below 30%.

desk verdict Useful first benchmark on EPIC-Kitchens with released models, but the headline comparison is built on single-run accuracy gaps that need error bars before they can be taken as fact. read the letter →

arxiv 1908.00867 v1 pith:PGE4UACT submitted 2019-08-02 cs.CV

Will Price , Dima Damen This is my paper · ORCID
classification cs.CV
keywords EPIC-Kitchensactionrecognitionegocentricvisiontemporalreasoningsegmentnetworksrelationalshiftmodulelong-tailclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper benchmarks three contemporary action-recognition architectures—TSN, TRN, and TSM—on EPIC-Kitchens, an egocentric video dataset of daily kitchen activities recorded in real homes. It aims to show how well these models handle the dataset's distinctive obstacles: a long-tailed distribution of verbs and nouns, actions in unseen kitchens, and the separate tasks of classifying verbs, nouns, and verb-noun actions. The central finding is that temporal-reasoning models (TSM and M-TRN) outperform the frame-aggregating baseline TSN on verbs and actions, which the authors take as evidence that inter-segment temporal reasoning is necessary for these tasks. Even so, the best action top-1 accuracy is 29.9% on seen kitchens and 17.9% on unseen kitchens, so the paper's overall message is that the problem remains open. The authors release all trained models so the reported numbers can serve as reproducible baselines.

What carries the argument

The load-bearing device is the uniform evaluation protocol: every model receives snippets sampled by the TSN strategy (n equal clip segments, one random snippet per segment) and is trained for 80 epochs with the same backbone options and an averaged softmax loss over separate verb and noun heads. The architectural mechanisms being compared are TSN's average/max pooling of per-segment class scores, TRN's relational module that computes order-sensitive relations over sets of segment features (multi-scale in M-TRN), and TSM's temporal shift module, which shifts a fraction of filter responses along the temporal dimension so later convolutions can mix information across segments. The comparison isolates temporal modelling because all models share the same sampling, backbone (ResNet-50 or BN-Inception), and training schedule.

What would settle it

Take TSN and retrain it on EPIC-Kitchens with a much larger search over learning rate, epoch count, and regularization, using several seeds; if a tuned TSN matches or beats M-TRN and TSM, the paper's central conclusion that temporal reasoning is required would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that on EPIC-Kitchens, models with explicit temporal reasoning—TSM's temporal shift of filter responses and M-TRN's order-sensitive relational module—achieve clearly higher verb and action classification accuracy than TSN, which only averages segment-level scores. The report documents the best observed action accuracy of 29.9% (seen) and 17.9% (unseen) with RGB-flow fusion, and notes that flow models are more resilient to the seen-to-unseen domain gap, dropping on average 22% relative in top-1 action accuracy versus 44% for RGB models. Noun classification depends less on temporal modelling, with TSN and TSM leading. The paper also shows that using the training set's verb-noun co-occurrence prior with Laplace smoothing adds roughly one percentage point of top-1 action accuracy on the unseen test set. These results are read as evidence that temporal reasoning is necessary but far from sufficient for egocentric action recognition.

Load-bearing premise

The comparison assumes that one shared training recipe—80 epochs, learning rates chosen only for stability, and no early stopping or repeated runs—gives every architecture a fair chance to show its best performance; if TSN simply needs more tuning, the temporal-reasoning gap could be an artifact.

Editorial extensions

If this is right

  • On EPIC-Kitchens, models that can relate segments over time are necessary for verb and action classification; TSN's segment averaging is not enough.
  • Fusing RGB and optical flow late in the network improves all tasks, and flow contributes the most to unseen-kitchen robustness (22% average relative drop for flow vs 44% for RGB on top-1 action).
  • The best action-recognition top-1 accuracy of 29.9% on seen kitchens and 17.9% on unseen kitchens means the dataset remains largely unsolved.
  • Applying the training-set verb-noun prior with Laplace smoothing adds roughly 1 percentage point on unseen action accuracy but little on seen, so the gain is mostly in generalization.
  • The released pretrained models provide reproducible baselines for follow-up work on egocentric action recognition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a natural follow-up would be a modular system that pairs flow-based verb classification with an object-centric noun classifier, since flow degrades far less on unseen kitchens and nouns are better recognized from RGB.
  • Editorial inference: because the paper's action scores are products of independent verb and noun probabilities, a joint action head or explicit verb-noun compatibility model could shift the ordering of TSM and M-TRN; the current ranking is only established under that independence assumption.
  • Editorial inference: TSM's drop when moving from 8 to 16 RGB segments, unlike its improvement with flow, suggests the segment-count result is not a simple 'more frames is better' story; testing with stronger regularization or temporal augmentation would clarify whether the effect is overfitting or an architectural quirk.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. This paper presents a benchmark evaluation of three contemporary action recognition models (TSN, TRN, and TSM, including multi-scale TRN) on the EPIC-Kitchens dataset. The authors train all models under a common protocol, evaluate verb, noun, and action classification on both seen and unseen kitchens, and analyze the effects of backbone choice, input modality, fusion, temporal segment count, and action priors. They report that TSM is competitive with M-TRN and that both outperform TSN, interpreting this as evidence that temporal reasoning is necessary for action recognition on this dataset. The best reported top-1 action accuracy is 29.9% on seen kitchens and 17.9% on unseen kitchens. The paper also releases pretrained models on GitHub.

Significance. If the comparative conclusions hold, this is a useful benchmark contribution for the egocentric video community: it provides a uniform comparison of widely used temporal architectures on a challenging dataset with a long-tailed class distribution and a train/test domain gap, and the released pretrained models have concrete reuse value. The paper's analysis of long-tail confusions and the seen/unseen performance drop is informative and goes beyond a bare accuracy table. However, the central comparative claim currently rests on single-run accuracy differences of only 2.0-2.5 points, with no error bars or multiple seeds, and with learning rates chosen only to ensure convergence. The manuscript therefore needs additional statistical grounding or appropriately qualified conclusions before the headline claim can be regarded as established.

major comments (1)
  1. [§3.2 Temporal support / Fig. 3] The paper states that the TSM performance drop at 16 segments was 'consistently observed across varying LRs,' but no numbers, learning-rate values, or seeds are provided for this robustness check. This check also does not address the more consequential single-run issue for the headline TSN-versus-temporal-model comparisons. Please either report the supporting data or soften the claim to reflect that it was observed informally.
minor comments (4)
  1. [§2] The TRN paragraph begins with 'TRN) [14]' which appears to be a typo for 'TRN [14]'.
  2. [§3.2] The sentence 'We hypothesis this is due to the absence of appearance information...' should read 'hypothesize'.
  3. [Table 3] The sentence 'The complexity of the models using ResNet-50 backbone is compared in Table 3,' ends with a comma and is grammatically incomplete.
  4. [§4 Released Models] The release description would benefit from stating the license, the exact optical-flow algorithm used to generate flow inputs, and the expected input preprocessing, so that other researchers can reproduce or extend the released models without ambiguity.

Circularity Check

0 steps flagged · score 1.0 of 10

Benchmark report with no circular derivation; the one self-citation (the authors' own EPIC-Kitchens dataset) is not load-bearing.

full rationale

This paper is an empirical benchmark, not a derivation, so most circularity patterns do not apply. The central claim that TSM is competitive with M-TRN and both outperform TSN rests on measured top-1 accuracies in Table 1 against fixed EPIC-Kitchens test labels and on previously published model definitions (TSN [12], TRN [14], TSM [7]), all external to this report. The dataset [1] is co-authored by the present authors, but the evaluation uses its official splits and ground truth; citing the dataset is a necessary description, not an argument that supplies the result. The action-prior formula in Eq. (2) is imported from [13] and then evaluated on held-out test sets, so it is not a fitted parameter renamed as a prediction. The only potentially questionable element is experimental: Section 3.1 states learning rates were 'the maximum we could achieve whilst maintaining convergence' with a single 80-epoch run per configuration, and Section 5's conclusion that temporal reasoning is necessary is based on 2-2.5 point top-1 gaps in Table 1. That is a tuning and statistical robustness concern, not circularity: the model ranking is measured against external ground truth and could be overturned by better hyperparameters or multiple seeds. No self-definitional, fitted-input, uniqueness-imported, or ansatz-by-citation step is present. Minor self-citation gives score 1 rather than 0, but there is no circular reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on a dataset the authors created and on standard hyperparameters; no new physical or mathematical entities are introduced.

free parameters (3)
  • Laplace smoothing coefficient for action priors = not reported
    Used in Eq. 2 to smooth verb-noun co-occurrence counts. The value affects the +0.8 to +1.4 point improvements on S2 in Table 2 and is not stated, so those numbers cannot be independently reproduced.
  • Learning rates = 0.01 for ResNet-50 RGB; 0.001 for BN-Inception RGB and all optical flow models
    Chosen as the maximum that maintains convergence (Section 3.1). Different choices could change the relative standing of the models, which is the paper's central claim.
  • Training epochs = 80
    Fixed budget without validation-based early stopping; models may be at different convergence points, which could affect the comparison.
assumptions (4)
  • domain assumption EPIC-Kitchens annotations and train/test splits are correct and representative
    The benchmark's ground truth is taken as given from the dataset paper [1]; errors in labels would directly bias all reported accuracies.
  • domain assumption The implementations of TSN, TRN, and TSM are faithful to their original papers
    The comparison is only meaningful if each model is a correct instantiation of the cited architecture; any implementation bug would change rankings.
  • domain assumption ImageNet pretrained initializations transfer equally to egocentric kitchen video
    All models start from ImageNet weights; differences in how much each architecture benefits from this initialization could skew the comparison.
  • ad hoc to paper The uniform training regime (80 epochs, fixed LR decay, no validation-based early stopping) is sufficient and fair for all models
    LRs are chosen per model as the maximum that converges; there is no evidence that all models are at comparable optimization status.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Evaluation of Action Recognition Models on EPIC-Kitchens." pith.science (2026). https://pith.science/paper/PGE4UACT

@misc{pith2026190800867,
  author       = {Pith},
  title        = {Pith review of: An Evaluation of Action Recognition Models on EPIC-Kitchens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGE4UACT}},
  note         = {Machine review of arXiv:1908.00867}
}
read the original abstract

We benchmark contemporary action recognition models (TSN, TRN, and TSM) on the recently introduced EPIC-Kitchens dataset and release pretrained models on GitHub (https://github.com/epic-kitchens/action-models) for others to build upon. In contrast to popular action recognition datasets like Kinetics, Something-Something, UCF101, and HMDB51, EPIC-Kitchens is shot from an egocentric perspective and captures daily actions in-situ. In this report, we aim to understand how well these models can tackle the challenges present in this dataset, such as its long tail class distribution, unseen environment test set, and multiple tasks (verb, noun and, action classification). We discuss the models' shortcomings and avenues for future research.

Figures

Figures reproduced from arXiv: 1908.00867 by the authors.

Figure 1
Figure 1. Fusion models’ performance on top-20 most frequent classes in training. Classes are ordered from top to bottom [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Two examples demonstrating where models capable of temporal reasoning, TRN and TSM, improve over TSN. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Top-1 accuracy on the seen test set when varying number of segments (during both training/testing) for M-TRN [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 9 canonical work pages

  1. [1]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018

    2018

  2. [2]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz 2https://epic-kitchens.github.io/2019#challenges Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluati...

    work page 2019

  3. [3]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

    2016

  4. [4]

    Batch normalization: Accelerating deep network training by reducing internal co- variate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In ICML, 2015

    2015

  5. [5]

    The kinetics human action video dataset, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017

    2017

  6. [6]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recogni- tion. In ICCV, 2011

    work page 2011

  7. [7]

    Lin, Chuang

    J. Lin, Chuang. Gan, and S. Han. Temporal shift module for efficient video understanding. arXiv, 2018

    work page 2018

  8. [8]

    A simple neural network module for relational rea- soning

    Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational rea- soning. In NeurIPS, 2017

    work page 2017

Show all 14 references
  1. [9]

    Two-stream con- volutional networks for action recognition in videos

    Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In NeurIPS, 2014

    2014

  2. [10]

    UCF101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild, 2012

    2012

  3. [11]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015

    2015

  4. [12]

    L. Wang, Y . Xiong, Z. Wang, Y . Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks for action recog- nition in videos. TPAMI, 2019

    2019

  5. [13]

    Long-Term Feature Banks for Detailed Video Understanding

    Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim- ing He, Philipp Krähenbühl, and Ross Girshick. Long-Term Feature Banks for Detailed Video Understanding. In CVPR, 2019

    2019

  6. [14]

    B. Zhou, A. Andonian, A. Oliva, and A. Torralba. Temporal relational reasoning in videos. In ECCV, 2018. 6

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.