An End-to-End Two-Stream Network Based on RGB Flow and Representation Flow for Human Action Recognition

Ka-Chun Fung; Kin-man Lam; Song-Jiang Lai; Tian-Shan Liu; Tsun-Hin Cheung

REVIEW 4 major objections 5 minor 18 references

An End-to-End Two-Stream Network Based on RGB Flow and Representation Flow for Human Action Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing the optical-flow branch of a two-stream egocentric action recognition model with a differentiable representation flow layer preserves accuracy on three benchmarks while shrinking per-video prediction time from minutes to under…

desk verdict Real speedup, shaky accuracy: the Table 4/5 ablation inconsistency undermines the flow-layer selection, and HMDB tuning contaminates the headline accuracy. read the letter →

arxiv 2411.18002 v1 pith:D6OCUYOG submitted 2024-11-27 cs.CV cs.AI

Song-Jiang Lai , Tsun-Hin Cheung , Ka-Chun Fung , Tian-Shan Liu , Kin-Man Lam This is my paper · ORCID

classification cs.CVcs.AI

keywords two-streamnetworkegocentricactionrecognitionrepresentationflowopticalreplacementclassactivationmapConvLSTMend-to-endtrainingvideo

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing the optical flow branch of a two-stream egocentric action recognition network with a differentiable representation flow layer, enabling end-to-end training. The resulting model matches the original accuracy on GTEA61 (69.84%) and improves it by 0.65 and 0.84 percentage points on EGTEA GAZE+ and HMDB, respectively. Per-video prediction time drops from 101.68–203.99 seconds to 0.15–0.19 seconds. The authors argue this makes accurate two-stream action recognition practical for egocentric video.

What carries the argument

The representation flow layer is a fully differentiable motion estimator that computes a flow field $u$ by iteratively solving a total-variation energy minimization on deep feature maps, using learnable parameters rather than fixed hyperparameters. The iteration alternates between updating the flow estimate and the dual vector field $p$, with Sobel-filtered gradients of the feature maps as the data term. Because the layer is differentiable, the entire two-stream network can be trained end-to-end, and at inference time it eliminates the expensive step of computing and storing optical flow from raw video.

What would settle it

A held-out egocentric dataset with fast camera motion would separate the two motion representations: if the representation-flow branch trails the optical-flow branch by more than a few absolute accuracy points there, the claim that learned flow generalizes as well as classical flow for egocentric action is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that a two-stream network for egocentric action recognition can replace precomputed optical flow with a learnable representation flow layer operating on deep feature maps, without sacrificing accuracy and while gaining a three-orders-of-magnitude speedup. The hybrid model uses class activation maps to guide spatial attention and a ConvLSTM for temporal encoding in the RGB stream, and it stacks two representation flow layers separated by a convolutional layer in the motion stream. After joint training, the model attains 69.84% on GTEA61 (identical to the optical-flow baseline), 61.09% average accuracy on EGTEA GAZE+ (versus 60.44%), and 50.71% on HMDB (versus 49.87%), with inference times of 0.1881 s, 0.1503 s, and 0.1459 s per video respectively.

Load-bearing premise

The load-bearing premise is that the representation flow algorithm, originally designed for third-person action recognition, transfers effectively to egocentric action recognition inside the base two-stream model, and that the hyperparameters chosen on HMDB (two flow layers, 20 iterations, ResNet-34) remain optimal on the egocentric datasets without re-tuning.

Editorial extensions

If this is right

Two-stream action recognition for egocentric video no longer needs a separate optical-flow preprocessing stage, so models can be trained and deployed directly on raw RGB input.
Per-video prediction time falls from the order of minutes to well under a second, making real-time egocentric activity analysis feasible.
The accuracy gains on EGTEA GAZE+ and HMDB suggest representation flow can capture motion cues that are at least as informative as classical optical flow for a range of action types.
The two-branch design with joint training by cross-entropy loss is general; the same replacement could be applied to other two-stream architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The transferability assumption is the riskiest part: the representation flow hyperparameters were tuned on HMDB, and the paper does not report tuning on egocentric data, so a re-tuning on GTEA61 or EGTEA GAZE+ might change the reported comparisons.
The large runtime reduction suggests that for practical systems, the main bottleneck was precomputed optical flow, not the recognition network itself; removing it could enable on-device egocentric action recognition.
A direct comparison against an optical-flow branch with the same ResNet-34 backbone and no other architectural changes would isolate the effect of the flow representation more cleanly than the current comparison against the original model.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the runtime reduction is real and welcome, but the accuracy claim is not as solid as the abstract suggests. The internal inconsistency between Table 4 and Table 5 is load-bearing.

What's actually new: the paper replaces the optical flow branch of EgoRCNN with representation flow, making the whole thing trainable end-to-end, and measures the speed/accuracy trade-off on GTEA61, EGTEA GAZE+, and HMDB. It's a straightforward engineering combination, not a new method, but a legitimate practical variant.

What the paper does well: the speedup numbers are credible and well-motivated. Removing precomputed optical flow obviously eliminates the dominant cost, and the reported classification times are on the right order. The method section is clear enough to replicate the architecture, and comparing against the original EgoRCNN on the same datasets is the correct evaluation setup.

The soft spots are real. Table 5 reports 20.08% for the 'without representation flow layer' configuration, while Table 4 reports 50.00% for the RGB-only branch. Those numbers describe the same thing and are 30 points apart. That makes the ablation selecting '2 flow layers' unreliable, and the 4.26-point gain attributed to adding flow layers is likely an artifact of an unreported protocol change or a typo. On top of that, HMDB is used to select hyperparameters (flow layers, iterations, backbone), then the same split is used to report the headline 50.71% accuracy, so the 0.84% gain over the optical-flow baseline is not out-of-sample. There are no error bars or significance tests, and the exact match of 69.84% on GTEA61 adds to the suspicion that the accuracy differences are within noise.

None of this undercuts the efficiency story. The paper is worth a referee's time because the speedup is real and the comparison is honest in intent, but the accuracy claims need major revision: fix the ablation, add variance, and separate tuning from evaluation.

Recommendation: send to peer review with a request for major revision. The reader's conditional verdict is right; I lean skeptical on the accuracy numbers but fully accept the speedup.

Referee Report

4 major / 5 minor

Summary. The paper proposes an end-to-end two-stream action recognition network that replaces the optical-flow branch of the EgoRCNN model with a differentiable representation-flow branch. The RGB branch uses ResNet-34 with CAM-based spatial attention and ConvLSTM temporal encoding; the flow branch uses representation flow layers followed by a convolutional layer. On GTEA61, EGTEA GAZE+, and HMDB, the authors report that the proposed model matches or slightly exceeds the optical-flow two-stream baseline (69.84%, 61.09% vs 60.44%, 50.71% vs 49.87%) while reducing per-video prediction time from tens or hundreds of seconds to roughly 0.15-0.19 seconds. Ablation studies on HMDB select two flow layers, 20 iterations, and ResNet-34 as the flow backbone.

Significance. The central speed claim is credible and potentially valuable: removing offline optical-flow extraction is a well-motivated way to make two-stream egocentric action recognition practical, and the reported runtime reduction (roughly three orders of magnitude) follows directly from eliminating a dominant pre-processing cost. The architecture itself is a straightforward integration of published components (representation flow from Piergiovanni and Ryoo; EgoRCNN from Sudhakaran and Lanz), and the paper does not claim a new theory or a new dataset. If the accuracy parity/improvement were established with proper statistical care, the contribution would be a useful engineering result for real-time egocentric recognition. However, the accuracy evidence is not currently reliable enough to support the headline claim of exceeding the optical-flow baseline, because the ablation that motivates the key design choices is internally inconsistent and the main comparison is performed on the same HMDB split used for hyperparameter selection.

major comments (4)

[Section III-C, Tables 4 and 5] The ablation that selects the number of representation flow layers is internally inconsistent with the main results. Table 4 reports RGB flow only at 50.00% on HMDB and representation flow only at 24.34%; Table 5 reports 'Without representation flow layer' at 20.08%. If this row means the flow branch is the RGB-only model, it should match 50.00%; if it means something else, such as a model with the flow layers removed but other components retained, the configuration is not defined and is 30 points below the RGB-only baseline. Either way, the 4.26-point gain from adding two flow layers (20.08 to 24.34) is not a controlled comparison, so the choice of two flow layers is not supported by the evidence presented.
[Section III-B, Table 4 and Section III-C] The HMDB hyperparameters (number of flow layers, iterations, and backbone size) are selected by ablations on HMDB and then the same HMDB results are used to claim a 0.84% improvement over the optical-flow baseline. This makes the reported HMDB accuracy an in-sample, hyperparameter-tuned number rather than an out-of-sample evaluation. A held-out validation split or a separate test set is needed before the accuracy improvement on HMDB can be credited.
[Section III-B, Tables 1-4] All accuracy comparisons are reported as point estimates without error bars, multiple seeds, or significance testing. On GTEA61 the proposed and baseline models both report exactly 69.84%, and the claimed gains on EGTEA GAZE+ (0.65%) and HMDB (0.84%) are well within the range that could arise from random seed variation, especially for small datasets. The paper should report the variance across runs or demonstrate that the differences are statistically meaningful before claiming that the proposed model 'exceeds' the original model.
[Section III-C, Tables 8-9] The state-of-the-art comparison is not usable as presented. The text states that Table 8 shows GTEA61 results and Table 9 shows EGTEA GAZE+ results, but the tables are not reproduced with their numeric entries, and the text claims 'a significant improvement' without giving the compared accuracies. The comparison should either be removed or presented with concrete numbers, splits, and citations.

minor comments (5)

[Abstract and Keywords] The keyword list contains 'CAM' twice; the duplicate should be removed.
[Section I, references] The survey of prior work attributes 'Heng et al.' to dense trajectories, but the correct author is Heng Wang; the reference list itself uses 'Wang, Heng' correctly. Consider fixing the in-text name.
[Section II-B, Equation (18)] The cross-entropy loss sums over k categories with indicator (c == i), but the notation for the summation index and the use of k are not defined consistently; please clarify the index range and the meaning of k.
[Section III-B, GTEA61 paragraph] There are typographical issues such as 'pre video' and 'state-of-art'; these should be corrected during revision.
[Tables 1-4] Some table captions are incomplete or generic ('Table 1: Comparison ...') and the reader must interpolate the configuration from the text; adding explicit column headers and dataset split information inside each table would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: measured empirical comparison whose central accuracy and runtime claims do not reduce to model inputs.

full rationale

This is an empirical comparison paper rather than a derivation. The representation flow layer is imported from Piergiovanni et al. [11], and the RGB-stream/attention architecture is imported from Sudhakaran et al. [12]; neither is a self-citation, and both are treated as external building blocks. The reported accuracy numbers (GTEA61 69.84%, EGTEA GAZE+ 61.09%, HMDB 50.71%) and runtimes (0.15-0.19 s) are measured on fixed dataset splits, not derived from the model's own equations. The CAM-based attention in Eqs. (1)-(2) uses the winning class prediction to build a saliency map and re-weight features; this is a standard self-referential attention mechanism and does not define the reported accuracy in terms of itself. No equation in the paper sets an output equal to its own input, and no fitted parameter is relabeled as a prediction. Two validity concerns are noted but are not circularity: the hyperparameters in Tables 5-7 are tuned on HMDB and then the headline HMDB comparison in Table 4 is reported on the same set (an in-sample selection issue), and Table 5's 'Without representation flow layer' row (20.08%) conflicts with Table 4's RGB-only row (50.00%), suggesting an unreported protocol difference. These are correctness risks, not circular reasoning.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; it combines existing algorithms. The central claim rests on standard domain assumptions about the transferability of representation flow and the CAM attention mechanism, plus hyperparameters selected on the HMDB test set.

free parameters (5)

number_of_representation_flow_layers = 2
Selected by ablation on HMDB (Table 5); 0, 1, and 3 layers yield lower accuracy.
flow_iterations = 20
Selected by ablation on HMDB (Table 6); 10, 30, and 50 iterations yield lower accuracy.
flow_branch_backbone = ResNet-34
Selected by ablation on HMDB (Table 7); ResNet-18, ResNet-50, and ResNet-101 yield lower accuracy.
input_clip_length = 16 frames
Set 'for optimal performance' without supporting ablation evidence (Section III-A).
convLSTM_hidden_units = 512
Chosen by hand for the RGB branch (Section III-A); no ablation is shown.

assumptions (4)

domain assumption Representation flow algorithm from [11] is effective as a motion representation branch for egocentric action recognition.
The paper replaces optical flow with representation flow without re-deriving or adapting the algorithm; transfer is assumed (Section II-C, II-E).
domain assumption Two-stage training and decision-level fusion from EgoRCNN [12] remain valid when the optical flow branch is replaced.
The proposed model follows the original training protocol (Section III-A) and fuses branches at decision level; no analysis of interaction is given.
domain assumption CAM-based spatial attention using the winning class improves recognition.
Equation (2) multiplies features by the softmax of the CAM of the winning class; no ablation isolates this component's contribution.
ad hoc to paper HMDB hyperparameter choices (Tables 5-7) generalize to GTEA61 and EGTEA GAZE+.
The same settings are used across all datasets; no per-dataset tuning or validation is described.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An End-to-End Two-Stream Network Based on RGB Flow and Representation Flow for Human Action Recognition." pith.science (2026). https://pith.science/paper/D6OCUYOG

@misc{pith2026241118002,
  author       = {Pith},
  title        = {Pith review of: An End-to-End Two-Stream Network Based on RGB Flow and Representation Flow for Human Action Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6OCUYOG}},
  note         = {Machine review of arXiv:2411.18002}
}

read the original abstract

With the rapid advancements in deep learning, computer vision tasks have seen significant improvements, making two-stream neural networks a popular focus for video based action recognition. Traditional models using RGB and optical flow streams achieve strong performance but at a high computational cost. To address this, we introduce a representation flow algorithm to replace the optical flow branch in the egocentric action recognition model, enabling end-to-end training while reducing computational cost and prediction time. Our model, designed for egocentric action recognition, uses class activation maps (CAMs) to improve accuracy and ConvLSTM for spatio temporal encoding with spatial attention. When evaluated on the GTEA61, EGTEA GAZE+, and HMDB datasets, our model matches the accuracy of the original model on GTEA61 and exceeds it by 0.65% and 0.84% on EGTEA GAZE+ and HMDB, respectively. Prediction runtimes are significantly reduced to 0.1881s, 0.1503s, and 0.1459s, compared to the original model's 101.6795s, 25.3799s, and 203.9958s. Ablation studies were also conducted to study the impact of different parameters on model performance. Keywords: two-stream, egocentric, action recognition, CAM, representation flow, CAM, ConvLSTM

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

[1]

On space -time interest points

Laptev, Ivan. "On space -time interest points." International journal of computer vision 64 (2005): 107-123

work page 2005
[2]

(2008, June)

Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008, June). Learning realistic human actions from movies. In 2008 IEEE conference on computer vision and pattern recognition (pp. 1-8). IEEE

work page 2008
[3]

Action recognition with improved trajectories

Wang, Heng, and Cordelia Schmid. "Action recognition with improved trajectories." Proceedings of the IEEE international conference on computer vision. 2013

work page 2013
[4]

Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei -Fei, L. (2014). Large -scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 1725-1732)

work page 2014
[5]

Convolutional two -stream network fusion for video action recognition

Feichtenhofer, Christoph, Axel Pinz, and Andrew Zisserman. "Convolutional two -stream network fusion for video action recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016

work page 2016
[6]

Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015). Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4694-4702)

work page 2015
[7]

Temporal relational reasoning in videos

Zhou, Bolei, et al. "Temporal relational reasoning in videos." Proceedings of the European conference on computer vision (ECCV). 2018

work page 2018
[8]

(2017, February)

Song, S., Lan, C., Xing, J., Zeng, W., & Liu, J. (2017, February). An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In Proceedings of the AAAI conference on artificial intelligence (Vol. 31, No. 1)

work page 2017

Show all 18 references

[9]

Fan, L., Huang, W., Gan, C., Ermon, S., Gong, B., & Huang, J. (2018). End -to-end learning of motion representation for video understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6016-6025

2018
[10]

Optical flow guided feature: A fast and robust motion representation for video action recognition

Sun, Shuyang, et al. "Optical flow guided feature: A fast and robust motion representation for video action recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018

2018
[11]

Representation flow for action recognition

Piergiovanni, A. J., and Michael S. Ryoo. "Representation flow for action recognition." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019

2019
[12]

Attention is all we need: Nailing down object -centric attention for egocentric activity recognition

Sudhakaran, Swathikiran, and Oswald Lanz. "Attention is all we need: Nailing down object -centric attention for egocentric activity recognition." arXiv preprint arXiv:1807.11794 (2018)

2018 arXiv
[13]

Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2921-2929)

2016
[14]

Delving into egocentric actions

Li, Yin, Zhefan Ye, and James M. Rehg. "Delving into egocentric actions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015

2015
[15]

Going deeper into first -person activity recognition

Ma, Minghuang, Haoqi Fan, and Kris M. Kitani. "Going deeper into first -person activity recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016

2016
[16]

First person action recognition using deep learned descriptors

Singh, Suriya, Chetan Arora, and C. V. Jawahar. "First person action recognition using deep learned descriptors." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016

2016
[17]

Quo vadis, action recognition? a new model and the kinetics dataset

Carreira, Joao, and Andrew Zisserman. "Quo vadis, action recognition? a new model and the kinetics dataset." proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017

2017
[18]

Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2016). Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision (pp. 20-36). Springer, Cham

2016

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

On space -time interest points

Laptev, Ivan. "On space -time interest points." International journal of computer vision 64 (2005): 107-123

work page 2005

[2] [2]

(2008, June)

Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008, June). Learning realistic human actions from movies. In 2008 IEEE conference on computer vision and pattern recognition (pp. 1-8). IEEE

work page 2008

[3] [3]

Action recognition with improved trajectories

Wang, Heng, and Cordelia Schmid. "Action recognition with improved trajectories." Proceedings of the IEEE international conference on computer vision. 2013

work page 2013

[4] [4]

Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei -Fei, L. (2014). Large -scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 1725-1732)

work page 2014

[5] [5]

Convolutional two -stream network fusion for video action recognition

Feichtenhofer, Christoph, Axel Pinz, and Andrew Zisserman. "Convolutional two -stream network fusion for video action recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016

work page 2016

[6] [6]

Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015). Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4694-4702)

work page 2015

[7] [7]

Temporal relational reasoning in videos

Zhou, Bolei, et al. "Temporal relational reasoning in videos." Proceedings of the European conference on computer vision (ECCV). 2018

work page 2018

[8] [8]

(2017, February)

Song, S., Lan, C., Xing, J., Zeng, W., & Liu, J. (2017, February). An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In Proceedings of the AAAI conference on artificial intelligence (Vol. 31, No. 1)

work page 2017