REVIEW 4 major objections 6 minor 39 references
Action Recognition in Untrimmed Videos with Composite Self-Attention Two-Stream Framework
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read One composite two-stream framework reaches 98.7% on UCF101 and 89.7% on THUMOS14 and supports zero-shot action recognition.
desk verdict Central zero-shot claim is entirely untested, and the attention equations don't type-check; the reported numbers don't rescue it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the composite two-stream architecture, with graph network models adopted in both branches. A classifier branch uses pre-trained word vectors and a knowledge graph of everyday concepts inside a graph convolutional network (a network that passes messages along a graph of action relations) to relate seen and unseen action classes. A composite feature branch runs two sub-streams, one for trimmed video and one for untrimmed video, each containing a three-channel self-attention model that maps extracted spatial and temporal feature matrices to attention-weighted matrices; the per-channel operations differ only in activation function (sigmoid, tanh, Leaky ReLU). A transfer module aligns the two sub-streams' classifier parameters through generalized maximum mean discrepancy, and a regularization loss enforces smooth, sparse attention weights. This combined machinery is what the paper credits with focusing on key frames in untrimmed footage while retaining semantic generalization for zero-shot categories.
What would settle it
Implement the self-attention equations of Section 3.1 with the stated shapes: a G×1 attention vector applied to an s×G feature matrix cannot produce an s×G output, so the model as written does not have a well-defined forward pass; a reader can check the released code or a clarified derivation for the missing operation and then verify whether it reproduces 98.7% on UCF101 and 89.7% on THUMOS14. Any implementation that silently patches the dimension mismatch would make the reported numbers unreproducible from the paper's own definitions.
Extended reading notes
Core claim
The central claim, stated on the paper's own terms, is that a multi-channel self-attention mechanism can weight video frames from several complementary perspectives and, embedded in a two-stream transfer framework, improve action recognition on both trimmed and untrimmed video. Three attention channels, using sigmoid, tanh, and Leaky ReLU activations through successive fully connected layers and softmax, each produce a set of frame weights; the weighted spatial and temporal features are then fed to a transfer module that aligns a trimmed-video stream with an untrimmed-video stream using generalized maximum mean discrepancy. The paper also claims that replacing the attention module of a two-stream graph convolutional network with this composite attention module yields zero-shot action recognition in untrimmed videos. Empirically, the model reports 98.7% average accuracy on UCF101 and 89.7% on THUMOS14, above the listed baselines.
Load-bearing premise
The load-bearing premise is that the attention vectors, defined as G×1 columns, can be applied to an s×G feature matrix to return an s×G weighted matrix; as written this multiplication is dimensionally undefined, so every reported accuracy depends on the unstated operation that actually performs the weighting.
Editorial extensions
If this is right
- If the paper is right, a single model outperforms the listed two-stream, C3D, TSN, and UntrimmedNet baselines on both trimmed and untrimmed benchmarks.
- The transfer module implies that knowledge learned on labeled trimmed video can be reused for untrimmed video, reducing the need for exhaustive untrimmed annotation.
- The zero-shot design implies that novel action classes can be recognized from their semantic relations in a knowledge graph plus word vectors, with no positive video examples of those classes.
- Because each attention channel uses a distinct activation, the learned weights are in principle separable into different aspects, which could support analysis of which frames the model treats as key.
Reading between the lines
- An implication the authors leave implicit is that the reported gains may depend on the pretrained backbone features as much as on the attention module; a controlled comparison against a plain temporal-pooling baseline with the same features would separate the two.
- The zero-shot claim is not backed by a reported zero-shot accuracy in the paper, so the fair reading is that zero-shot support is structural rather than measured; running the paper's TD, G, and TD+G splits and reporting accuracies would test it.
- The dimension mismatch in the written attention multiplication suggests the actual implementation likely uses a column-wise scaling or broadcast rather than a literal matrix product; identifying that operation matters because different choices would yield different feature weighting behaviors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes a composite two-stream framework for action recognition in untrimmed videos. The framework contains a classifier branch and a composite feature branch; each branch uses a graph network, and the composite feature branch uses a 3-channel self-attention mechanism to weight frames in the spatial and temporal dimensions. The authors claim the model supports zero-shot action recognition and reports 98.7% accuracy on UCF101 for trimmed action recognition and 89.7% accuracy on THUMOS14 for untrimmed action recognition. The evaluation section lists three tasks (TAR, UTAR, ZSAR), but only results for the first two are presented; no zero-shot experiment is reported.
Significance. If the claims were established, the architecture would combine multi-channel self-attention, transfer from trimmed to untrimmed video, and zero-shot recognition in a single framework, which would be a noteworthy contribution to action recognition. The paper also contains a detailed expression for an attention regularization loss in Section 3.2, and the choice of evaluation datasets (UCF101 and THUMOS14) is appropriate for the stated problem. However, the central zero-shot claim is not tested, no code or data-split details are released, and the attention equations in Section 3.1 contain a dimensionally undefined product that undermines the forward pass. As presented, the contribution is therefore not established.
major comments (4)
- [Sections 4.2 and 4.4] The abstract, title, and conclusion claim zero-shot action recognition for untrimmed videos, and Section 4.2 lists ZSAR as the third evaluation phase, but Section 4.4 reports no ZSAR accuracy, no comparison with the zero-shot baselines in [5, 19, 39], and no result for any of the TD, G, or TD+G splits defined in Tables 1-3. The central claim of the paper is therefore unsupported by any experiment.
- [Section 3.1] The attention multiplication is dimensionally undefined. V1j is stated to be in R^(G x 1) and is then said to be multiplied by the spatial feature matrix S in R^(s x G) to obtain the attentional matrix S_i in R^(s x G), but a G x 1 vector times an s x G matrix is not defined without an unstated transposition or column-scaling rule. The same problem occurs for V2j in R^((G-1) x 1) and T in R^(t x (G-1)). Because these attentional matrices enter the inference expression in Section 3.1 and the upper-stream expression in Section 3.2, the forward pass and all reported results depend on this unspecified operation.
- [Section 4.4 and Table 5] The UTAR comparison is not apples-to-apples. Section 4.2 states that classification accuracy is adopted as the metric, whereas UntrimmedNet [17] and most THUMOS14 untrimmed action recognition methods are evaluated using mean average precision (mAP) under a detection or threshold protocol. No mAP value or protocol details are given for the proposed method, so the 89.7% figure in Table 5 cannot be compared with the listed baselines.
- [Section 3.3] The zero-shot extension is described in a single paragraph that says the attention module of [5] is replaced by the composite self-attention model. No equations or algorithmic steps show how the classifier branch's GloVe encoding and ConceptNet knowledge graph combine with the composite feature branch to predict unseen classes. Independent of the missing ZSAR experiments, there is no described mechanism that would justify the zero-shot generalization claim.
minor comments (6)
- [Section 4.4] The text refers to "Table1" and "Table2" when discussing TAR and UTAR, but the relevant tables are labeled Table 4 and Table 5; the table captions should also state the evaluation metric used.
- [Section 3.1] The word "sigmod" appears repeatedly and should be "sigmoid"; similarly, "THOUMOS14" in Section 4 should be "THUMOS14."
- [Section 3.2] In the list of transferred classifier parameters, "P23, P22, P23" should presumably be "P21, P22, P23"; the current text omits P21 and duplicates P23.
- [Section 3.3] The heading of Section 3.3 duplicates the heading of Section 3.2; it should describe the composite two-stream zero-shot model.
- [Section 3.1] The phrases "we named this first frame" and "this second frame" should read "first branch" and "second branch" or "first extractor" and "second extractor," because the text is describing feature extractors, not frames.
- [Section 3.2] The labels "loss1, inference" and "loss2, inference" denote inference expressions rather than training losses; the actual training objectives for loss1 and loss2 are not explicitly defined beyond a mention of cross-entropy.
Circularity Check
No circularity: the framework is a borrowed-component architecture with an untested zero-shot claim, but no prediction reduces to its inputs.
full rationale
No load-bearing step in the paper reduces by construction to its own inputs. The multi-channel self-attention module is defined as a trained feature-weighting operation, and the reported results are standard classification accuracies on UCF101 and THUMOS14; no target metric is embedded in the loss or in the attention definitions. The composite feature branch is trained with a cross-entropy loss (loss2), an MMD transfer loss (loss3), and an attention regularization term (loss4), and the classifier parameters are transferred from a trimmed-video stream to an untrimmed-video stream; this is a conventional transfer-learning setup rather than a fit renamed as a prediction. The zero-shot portion of the paper is asserted by replacing the attention module of the external reference [5], which is a citation to other authors' work, not a self-citation, and no zero-shot experiment is reported; that is an evidential gap, not circularity. The dimension mismatch in Section 3.1, where the attention vector V1j is stated to be G x 1 and is then multiplied by the s x G matrix S to produce an s x G matrix, makes the forward pass formally undefined, but an undefined operation is a correctness problem rather than a circular dependence of the claimed result on its inputs. The paper also does not invoke a uniqueness theorem or derive its architecture from first principles, so there is no self-referential derivation chain to unwind. Borrowed components and untested claims can weaken the evidence, but they do not make the derivation circular.
Assumptions & free parameters
free parameters (3)
- attention dimension hyperparameters a and b
- number of attention channels =
3
- data split proportions =
80/20 seen/unseen
assumptions (3)
- domain assumption Pre-trained ResNet101 features extracted on Kinetics are sufficient for action recognition.
- domain assumption The MMD formulation from [35] aligns classifier parameters between trimmed and untrimmed streams.
- domain assumption ConceptNet knowledge graph and GloVe word embeddings contain the semantic relations needed for zero-shot classification.
Cite this review
Pith. "Pith review of Action Recognition in Untrimmed Videos with Composite Self-Attention Two-Stream Framework." pith.science (2026). https://pith.science/paper/MUDQP2H4
@misc{pith2026190804353,
author = {Pith},
title = {Pith review of: Action Recognition in Untrimmed Videos with Composite Self-Attention Two-Stream Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/MUDQP2H4}},
note = {Machine review of arXiv:1908.04353}
}
read the original abstract
With the rapid development of deep learning algorithms, action recognition in video has achieved many important research results. One issue in action recognition, Zero-Shot Action Recognition (ZSAR), has recently attracted considerable attention, which classify new categories without any positive examples. Another difficulty in action recognition is that untrimmed data may seriously affect model performance. We propose a composite two-stream framework with a pre-trained model. Our proposed framework includes a classifier branch and a composite feature branch. The graph network model is adopted in each of the two branches, which effectively improves the feature extraction and reasoning ability of the framework. In the composite feature branch, a 3-channel self-attention models are constructed to weight each frame in the video and give more attention to the key frames. Each self-attention models channel outputs a set of attention weights to focus on a particular aspect of the video, and a set of attention weights corresponds to a one-dimensional vector. The 3-channel self-attention models can evaluate key frames from multiple aspects, and the output sets of attention weight vectors form an attention matrix, which effectively enhances the attention of key frames with strong correlation of action. This model can implement action recognition under zero-shot conditions, and has good recognition performance for untrimmed video data. Experimental results on relevant data sets confirm the validity of our model.
Figures
Reference graph
Works this paper leans on
-
[38]
Zhang, X., Shi, H., Li, C ., Zheng, K., Zhu, X., and Duan, L.: Learning transferable self - attentive representations for action recognition in untrimmed videos with weak supervi- sion. In: AAAI (2019)
work page 2019
-
[5]
Gao, J., Zhu, T., and Xu, C.: I know the relationships: zero -shot action recognition via two-stream graph convolutional networks and knowledge graphs. In: AAAI (2019)
work page 2019
-
[17]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , CVPR (2017)
Wang, L., Xiong, Y., Lin, D., and Gool, L.: UntrimmedNets for weakly supervised action recognition and detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , CVPR (2017)
work page 2017
-
[1]
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017) 12
Carreira, J., and Zisserman, A.: Quo vadis, action recognition? A new model and the kinet- ics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017) 12
work page 2017
-
[2]
In: Proceedings of the IEEE International Conference on Computer Vision, ICCV (2015)
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., and Brox, T.: Flownet: Learning optical flow with convolutional net- works. In: Proceedings of the IEEE International Conference on Computer Vision, ICCV (2015)
work page 2015
-
[3]
E., Gong, B., and Huang, J.: End -to-end learning of motion representation for video understanding
Fan, L., Huang, W., Chuang Gan, S. E., Gong, B., and Huang, J.: End -to-end learning of motion representation for video understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
work page 2018
-
[4]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2016)
Feichtenhofer, C., Pinz, A., and Z isserman, A.: Convolutional two -stream network fusion for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2016)
work page 2016
-
[6]
Y.-H., Hausknecht, Vijayanarasimhan, M
Ng, J. Y.-H., Hausknecht, Vijayanarasimhan, M. S., Vinyals, O., Monga, R., and Toderici, G.: Beyond short snippets: Deepnetworks for video classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2015)
work page 2015
Show all 39 references
-
[7]
C., and Snoek, C
Jain, M., Gemert, J. C., and Snoek, C. G. M: What do 15000 object categories tell us about classifying and localizing actions? In: Proceedings of the IEEE Confe rence on Computer Vision and Pattern Recognition, CVPR (2015)
2015
-
[8]
Yu.: 3D convolutional neural networks for human action recognition
Ji, S., Xu, W., Yang, M., and K. Yu.: 3D convolutional neural networks for human action recognition. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 221–231(2013)
2013
-
[9]
Mishra, A., Verma, V.K., Reddy, M. S. K., S, A., Rai, P., and Mittal, A.: a generative ap- proach to zero-shot and few-shot action recognition. In: WACV (2018)
2018
-
[10]
S., and Dean, J
Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G. S., and Dean, J. : Zero -shot learning by convex combination of semantic embeddings. In: ICLR (2014)
2014
-
[11]
In: Advances in Neural Information Processing Systems (NIPS), pp
Simonyan, K., and Zisserman A.: Two -stream convolutional networks for action recogni- tion in videos. In: Advances in Neural Information Processing Systems (NIPS), pp. 568 – 576 (2014)
2014
-
[12]
In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2018)
Song, J., Shen, C., Yang, Y., Liu, Y., and Song, M.: Transductive unbiased embedding for zero-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2018)
2018
-
[13]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
Sun, S., Kuang, Z., Sheng, L., Ouyang, W., and Zhang, W.: Optical flow guided feature: A fast and robust motion representation for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
2018
-
[14]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2015)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2015)
2015
-
[15]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M.: A cl oser look at spa- tiotemporal convolutions for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2018)
2018
-
[16]
In: ICCV, pp
Wang, H., and Schmid, C.: Action recognition with improved trajectories. In: ICCV, pp. 3551–3558 (2013)
2013
-
[18]
In: ECCV, pp
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Van Gool, L.: Temporal segment networks: Towards good practices for deep action recognition. In: ECCV, pp. 20 – 36 (2016). 13
2016
-
[19]
In: ECCV (2016)
Xu, X., Hospedales, T., and Gong, S.: Multi -Task Zero-Shot Action Recognition with Pri- oritised Data Augmentation. In: ECCV (2016)
2016
-
[20]
In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2016)
Zhu, W., Hu, J., Sun, G., Cao, X., and Qiao, Y.: A key volume mining deep framework for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2016)
2016
-
[21]
D.: Depth2action: Exploring embedded depth for large -scale ac- tion recognition
Zhu, Y., and Newsam, S. D.: Depth2action: Exploring embedded depth for large -scale ac- tion recognition. In: ECCV Workshops, pp. 668–684 (2016)
2016
-
[22]
In: Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR (2018)
Zhu, Y., Long, Y., Guan, Y., Newsam, S., and Shao, L.: Towards universal representation for unseen action recognition. In: Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR (2018)
2018
-
[23]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2009)
Farhadi, A., Endres, I., Hoiem, D., and Forsyth, D.: Describing objects by their attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2009)
2009
-
[24]
H., Nickisch, H., and Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., and Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2009)
2009
-
[25]
In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2013)
Akata, Z., Perronnin, F ., Harchaoui, Z., and Schmid, C.: Label embedding for attribute - based classification. In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2013)
2013
-
[26]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017)
Morgado, P., and Vasconcelos, N.: Semantically consistent regularization f or zero -shot recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017)
2017
-
[27]
H., Schiele, B., and Akata, Z.: Zero -shot learning-a comprehensive evaluation of the good, the bad and the ugly
Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z.: Zero -shot learning-a comprehensive evaluation of the good, the bad and the ugly. In: IEEE Transactions on Pattern Analysis and Machine Intelligence. (2017)
2017
-
[28]
S., Shlens, J., Bengio, S., Dean, J., Mikolov , T., et al: Devise: A deep visual -semantic embedding model
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Mikolov , T., et al: Devise: A deep visual -semantic embedding model. In: Advances in neural information processing systems, pp. 2121–2129 (2013)
2013
-
[29]
In: P roceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2016)
Reed, S., Akata, Z., Lee, H., and Schiele, B.: Learning deep representations of fine-grained visual descriptions. In: P roceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR (2016)
2016
-
[30]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017)
Zhang, L., Xiang, T., and Gong, S.: Learning a deep embedding model for zero -shot learn- ing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017)
2017
-
[31]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017)
Qin, J., Liu, L., Shao, L., Shen, F., Ni, B., Chen, J., and Wang, Y.: Zero -shot action recog- nition with error -correcting output codes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017)
2017
-
[32]
C., Chen, C.-W., and Fei-Fei, L.: Modeling temporal structure of decomposable motion segments for activity classification
Niebles, J. C., Chen, C.-W., and Fei-Fei, L.: Modeling temporal structure of decomposable motion segments for activity classification. In: ECCV (2010)
2010
-
[33]
In: IEEE Trans
Pan, S.J., Yang, Q.: A survey on transfer learning. In: IEEE Trans. Knowl. Data Eng., 22(10), 1345-1359 (2010)
2010
-
[34]
In: International Conference on Machine Learning (2015)
Ganin, Y., and Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International Conference on Machine Learning (2015)
2015
-
[35]
I.: Learning transferable features with deep adaptation networks
Long, M., Cao, Y., Wang, J., and Jordan, M. I.: Learning transferable features with deep adaptation networks. In: International Conference on Machine Learning (2015)
2015
-
[36]
M., Rasch,M
Gretton, A., Borgwardt, K. M., Rasch,M. J., Scholkopf, B., and Smola, A. J.: A kernel two- sample test. In: JMLR(13), 723–773 (2012). 14
2012
-
[37]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR (2017)
Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T: Adversarial discriminative domain ad- aptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR (2017)
2017
-
[39]
International Journal of Computer Vision 123(3), 309-333(2017)
Xu, X., Hospedales, T., and Gong, S.: Transductive zero -shot action recognition by w ord- vector embedding. International Journal of Computer Vision 123(3), 309-333(2017)
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.