{"id":"420f8a1f-1ae8-41d7-aaf0-645b7f0ea318","arxiv_id":"1908.04353","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-stream framework with multi-channel self-attention is claimed to improve action recognition, but the zero-shot claim is not supported by any experimental results.","lead":"This paper proposes a composite two-stream network for action recognition in untrimmed videos, combining multi-channel self-attention with transfer learning from trimmed videos. It reports high accuracy on standard benchmarks, but the paper never presents results for the zero-shot scenario promised in the title and abstract.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot claim is untested and the attention dimension mismatch makes the forward pass ill-defined; rejection stands.","rationale":"The reader's weakest assumption (dimension mismatch in attention feature computation) is backed by the equations in Section 3.1, and the zero-shot-claim absence is verified by scanning Section 4: evaluation phases include TAR, UTAR, and ZSAR, but no ZSAR numbers are presented. Both issues are load-bearing because the first invalidates the forward pass and the second removes the paper's distinct contribution. I agree with the reader's verdict of REJECT, with no mitigating factors such as code release, machine-checked proofs, or reproducibility artifacts. The paper also contains internal inconsistencies (e.g., Section 3.3 titled identically to Section 3.2, P23/P22 typos), which reinforce but do not replace the two main concerns.","tokens_in":10564,"tokens_out":1389,"duration_ms":11673,"concrete_test":"Request or reconstruct the actual forward pass: (1) Check whether the paper's code or a runnable implementation is provided; if not, implement Equations in Section 3.1 with a fixed G and s and verify that S_i and T_i can be computed without adding a transposition or column-scaling. (2) For the zero-shot claim, run the TD/G/TD+G splits described in Section 4.1 (UCF101 train, THUMOS14 test) and report the ZSAR accuracy; if no such result can be produced, the abstract's zero-shot claim is unsupported. A single verification of these two items settles the concern.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central contribution, in its title, abstract, and conclusion, is zero-shot action recognition for untrimmed videos, yet Section 4 reports no zero-shot experiment at all: the three evaluation phases are described in 4.2, but only TAR (Table 4, UCF101) and UTAR (Table 5, THUMOS14) results appear, with no ZSAR table or accuracy. The zero-shot framework described in Section 3.3 is a one-paragraph hand-waved replacement of the attention module of [5], so there is no evidence the claimed 89.7% UTAR or 98.7% TAR transfers to unseen classes. Independently, Section 3.1 defines V1j in R^{G x 1} and then states that multiplying V1j by the spatial feature matrix S (s x G) yields S_i in R^{s x G}; the product V1j * S is dimensionally invalid (G x 1 times s x G), and the matching problem appears in the temporal branch (V2j in R^{(G-1) x 1} times T in R^{t x (G-1)}). Unless an unstated transposition or column-scaling is assumed, the attentional matrices S_i and T_i, and therefore the loss and inference expressions in Section 3.1 and 3.2, are undefined. The reader's concern is therefore load-bearing: both the headline contribution (zero-shot) and the core mechanism (attention weighting) lack a well-defined, tested implementation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a composite two-stream framework for action recognition in untrimmed videos. The framework contains a classifier branch and a composite feature branch; each branch uses a graph network, and the composite feature branch uses a 3-channel self-attention mechanism to weight frames in the spatial and temporal dimensions. The authors claim the model supports zero-shot action recognition and reports 98.7% accuracy on UCF101 for trimmed action recognition and 89.7% accuracy on THUMOS14 for untrimmed action recognition. The evaluation section lists three tasks (TAR, UTAR, ZSAR), but only results for the first two are presented; no zero-shot experiment is reported.","tokens_in":10850,"tokens_out":5112,"duration_ms":49293,"significance":"If the claims were established, the architecture would combine multi-channel self-attention, transfer from trimmed to untrimmed video, and zero-shot recognition in a single framework, which would be a noteworthy contribution to action recognition. The paper also contains a detailed expression for an attention regularization loss in Section 3.2, and the choice of evaluation datasets (UCF101 and THUMOS14) is appropriate for the stated problem. However, the central zero-shot claim is not tested, no code or data-split details are released, and the attention equations in Section 3.1 contain a dimensionally undefined product that undermines the forward pass. As presented, the contribution is therefore not established.","major_comments":[{"comment":"The abstract, title, and conclusion claim zero-shot action recognition for untrimmed videos, and Section 4.2 lists ZSAR as the third evaluation phase, but Section 4.4 reports no ZSAR accuracy, no comparison with the zero-shot baselines in [5, 19, 39], and no result for any of the TD, G, or TD+G splits defined in Tables 1-3. The central claim of the paper is therefore unsupported by any experiment.","section":"Sections 4.2 and 4.4"},{"comment":"The attention multiplication is dimensionally undefined. V1j is stated to be in R^(G x 1) and is then said to be multiplied by the spatial feature matrix S in R^(s x G) to obtain the attentional matrix S_i in R^(s x G), but a G x 1 vector times an s x G matrix is not defined without an unstated transposition or column-scaling rule. The same problem occurs for V2j in R^((G-1) x 1) and T in R^(t x (G-1)). Because these attentional matrices enter the inference expression in Section 3.1 and the upper-stream expression in Section 3.2, the forward pass and all reported results depend on this unspecified operation.","section":"Section 3.1"},{"comment":"The UTAR comparison is not apples-to-apples. Section 4.2 states that classification accuracy is adopted as the metric, whereas UntrimmedNet [17] and most THUMOS14 untrimmed action recognition methods are evaluated using mean average precision (mAP) under a detection or threshold protocol. No mAP value or protocol details are given for the proposed method, so the 89.7% figure in Table 5 cannot be compared with the listed baselines.","section":"Section 4.4 and Table 5"},{"comment":"The zero-shot extension is described in a single paragraph that says the attention module of [5] is replaced by the composite self-attention model. No equations or algorithmic steps show how the classifier branch's GloVe encoding and ConceptNet knowledge graph combine with the composite feature branch to predict unseen classes. Independent of the missing ZSAR experiments, there is no described mechanism that would justify the zero-shot generalization claim.","section":"Section 3.3"}],"minor_comments":[{"comment":"The text refers to \"Table1\" and \"Table2\" when discussing TAR and UTAR, but the relevant tables are labeled Table 4 and Table 5; the table captions should also state the evaluation metric used.","section":"Section 4.4"},{"comment":"The word \"sigmod\" appears repeatedly and should be \"sigmoid\"; similarly, \"THOUMOS14\" in Section 4 should be \"THUMOS14.\"","section":"Section 3.1"},{"comment":"In the list of transferred classifier parameters, \"P23, P22, P23\" should presumably be \"P21, P22, P23\"; the current text omits P21 and duplicates P23.","section":"Section 3.2"},{"comment":"The heading of Section 3.3 duplicates the heading of Section 3.2; it should describe the composite two-stream zero-shot model.","section":"Section 3.3"},{"comment":"The phrases \"we named this first frame\" and \"this second frame\" should read \"first branch\" and \"second branch\" or \"first extractor\" and \"second extractor,\" because the text is describing feature extractors, not frames.","section":"Section 3.1"},{"comment":"The labels \"loss1, inference\" and \"loss2, inference\" denote inference expressions rather than training losses; the actual training objectives for loss1 and loss2 are not explicitly defined beyond a mention of cross-entropy.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The manuscript's headline contribution is zero-shot action recognition for untrimmed videos, but no zero-shot experiment appears anywhere in the paper, and the core attention equations are dimensionally invalid. Given the absence of code, the inconsistent THUMOS14 protocol, and the untested central claim, I do not see a revision path within the current manuscript's scope that would preserve the present claims; a substantially rewritten paper with a well-defined forward pass and a complete ZSAR evaluation would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper claims zero-shot action recognition for untrimmed videos, but it contains no zero-shot experiments, and the core attention computation is dimensionally ill-defined. The reported 89.7% on THUMOS14 is not enough to save it.\n\nWhat is new here is not much. The architecture is a composite two-stream built from the self-attentive transfer branch of Zhang et al. [38] and the two-stream graph network of Gao et al. [5]. The only genuine variation is using three activation functions (sigmoid, tanh, Leaky ReLU) in parallel attention heads to get multiple attention vectors. That is a legitimate small idea. The paper also identifies a real problem—background frames in untrimmed videos—and it is honest about borrowing from [38] and [5]. The transfer loss via MMD is spelled out in more detail than one usually sees. I don't want to be unfair: the authors have a direction, and they state it plainly.\n\nThe problems are load-bearing. First, the central claim is not tested. Section 4.2 lists three evaluation phases—TAR, UTAR, and ZSAR—but the results section contains only TAR (UCF101) and UTAR (THUMOS14). There is no zero-shot accuracy anywhere. The abstract and conclusion say the model 'can implement action recognition under zero-shot conditions,' yet the paper never tries it. That alone is fatal for the paper's stated contribution.\n\nSecond, the attention equations do not type-check. Section 3.1 defines V1j as G×1 and says multiplying it by S (s×G) gives S_i in s×G. That product is undefined as written. To get s×G you would need to scale the columns of S by V, which is an element-wise operation, not a matrix product. The same issue appears in the temporal branch. Since the attention-weighted matrices S_i and T_i feed into the loss and inference expressions, the forward pass is under-specified. The stress-test note is correct on this point.\n\nThird, the THUMOS14 comparison is likely misleading. UntrimmedNet's 82.2 is an mAP number; the paper reports 89.7 as 'accuracy' and compares it to methods evaluated in accuracy. Without the proper protocol, the comparison is apples to oranges. The dataset statistics (2500/1574 untrimmed videos) also do not match standard THUMOS14, which suggests the evaluation pipeline is not well understood.\n\nThe prose is rough, but I would not reject on prose alone. The absence of zero-shot results, the undefined attention operation, and the questionable metric are enough. This is not a paper to send to peer review in its current state. If the authors later provide a correct derivation and actual ZSAR numbers on the standard splits, it might deserve another look. For now, desk reject.","headline":"Central zero-shot claim is entirely untested, and the attention equations don't type-check; the reported numbers don't rescue it.","tokens_in":11378,"tokens_out":3236,"would_cite":false,"duration_ms":30256,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One composite two-stream framework reaches 98.7% on UCF101 and 89.7% on THUMOS14 and supports zero-shot action recognition.","keywords":["action recognition","untrimmed videos","zero-shot action recognition","self-attention","graph convolutional network","two-stream network","transfer learning"],"falsifier":"Implement the self-attention equations of Section 3.1 with the stated shapes: a G×1 attention vector applied to an s×G feature matrix cannot produce an s×G output, so the model as written does not have a well-defined forward pass; a reader can check the released code or a clarified derivation for the missing operation and then verify whether it reproduces 98.7% on UCF101 and 89.7% on THUMOS14. Any implementation that silently patches the dimension mismatch would make the reported numbers unreproducible from the paper's own definitions.","tokens_in":10367,"feed_emoji":"🎬","tokens_out":13293,"duration_ms":109238,"temperature":0.7,"pith_summary":"One architecture, the paper argues, can handle both problems that usually need separate designs: untrimmed video with many background frames, and action classes with no positive training examples. It couples a classifier branch that reasons over an action-knowledge graph with a composite feature branch whose three self-attention channels each emphasize a different aspect of the frames. On the reported experiments the framework reaches 98.7% accuracy on UCF101 trimmed recognition and 89.7% on THUMOS14 untrimmed recognition, and the design is claimed to support zero-shot recognition of unseen classes. The intended contribution is a unified, transfer-friendly attention mechanism that improves both trimmed and untrimmed video understanding.","feed_headline":"Self-attention two-stream hits 98.7% on UCF101, 89.7% on THUMOS14","feed_subtitle":"Addresses both background-frame noise in long videos and classification of never-seen action classes.","key_machinery":"The carrying mechanism is the composite two-stream architecture, with graph network models adopted in both branches. A classifier branch uses pre-trained word vectors and a knowledge graph of everyday concepts inside a graph convolutional network (a network that passes messages along a graph of action relations) to relate seen and unseen action classes. A composite feature branch runs two sub-streams, one for trimmed video and one for untrimmed video, each containing a three-channel self-attention model that maps extracted spatial and temporal feature matrices to attention-weighted matrices; the per-channel operations differ only in activation function (sigmoid, tanh, Leaky ReLU). A transfer module aligns the two sub-streams' classifier parameters through generalized maximum mean discrepancy, and a regularization loss enforces smooth, sparse attention weights. This combined machinery is what the paper credits with focusing on key frames in untrimmed footage while retaining semantic generalization for zero-shot categories.","core_discovery":"The central claim, stated on the paper's own terms, is that a multi-channel self-attention mechanism can weight video frames from several complementary perspectives and, embedded in a two-stream transfer framework, improve action recognition on both trimmed and untrimmed video. Three attention channels, using sigmoid, tanh, and Leaky ReLU activations through successive fully connected layers and softmax, each produce a set of frame weights; the weighted spatial and temporal features are then fed to a transfer module that aligns a trimmed-video stream with an untrimmed-video stream using generalized maximum mean discrepancy. The paper also claims that replacing the attention module of a two-stream graph convolutional network with this composite attention module yields zero-shot action recognition in untrimmed videos. Empirically, the model reports 98.7% average accuracy on UCF101 and 89.7% on THUMOS14, above the listed baselines.","pith_inferences":["An implication the authors leave implicit is that the reported gains may depend on the pretrained backbone features as much as on the attention module; a controlled comparison against a plain temporal-pooling baseline with the same features would separate the two.","The zero-shot claim is not backed by a reported zero-shot accuracy in the paper, so the fair reading is that zero-shot support is structural rather than measured; running the paper's TD, G, and TD+G splits and reporting accuracies would test it.","The dimension mismatch in the written attention multiplication suggests the actual implementation likely uses a column-wise scaling or broadcast rather than a literal matrix product; identifying that operation matters because different choices would yield different feature weighting behaviors."],"forward_implications":["If the paper is right, a single model outperforms the listed two-stream, C3D, TSN, and UntrimmedNet baselines on both trimmed and untrimmed benchmarks.","The transfer module implies that knowledge learned on labeled trimmed video can be reused for untrimmed video, reducing the need for exhaustive untrimmed annotation.","The zero-shot design implies that novel action classes can be recognized from their semantic relations in a knowledge graph plus word vectors, with no positive video examples of those classes.","Because each attention channel uses a distinct activation, the learned weights are in principle separable into different aspects, which could support analysis of which frames the model treats as key."],"supporting_citations":[{"why":"supplies the two-stream graph convolutional zero-shot architecture whose attention module the paper replaces","marker":"[5]"},{"why":"supplies the composite feature branch and the self-attention design for untrimmed action recognition that the paper extends","marker":"[38]"},{"why":"provides the generalized maximum mean discrepancy used to align trimmed and untrimmed streams in the transfer module","marker":"[35]"},{"why":"provides the pretrained two-stream baseline and pretraining context that the reported results are compared against","marker":"[1]"},{"why":"defines the word-vector embedding approach and the data split the experiments follow","marker":"[39]"},{"why":"is the main untrimmed action recognition baseline the model claims to outperform","marker":"[17]"},{"why":"is the TSN baseline used for both trimmed and untrimmed comparisons","marker":"[18]"}],"fun_headline_variants":["Multi-channel attention nails zero-shot and untrimmed action recognition","Composite self-attention: 98.7% UCF101, 89.7% THUMOS14, zero-shot ready","Two-stream attention with three channels handles zero-shot and untrimmed video","Self-attention weights frames from multiple views for zero-shot action recognition","Zero-shot action recognition in untrimmed videos via composite attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the attention vectors, defined as G×1 columns, can be applied to an s×G feature matrix to return an s×G weighted matrix; as written this multiplication is dimensionally undefined, so every reported accuracy depends on the unstated operation that actually performs the weighting.","fun_headline_variants_meta":{"raw":{"variants":["Multi-channel attention nails zero-shot and untrimmed action recognition","Composite self-attention: 98.7% UCF101, 89.7% THUMOS14, zero-shot ready","Two-stream attention with three channels handles zero-shot and untrimmed video","Self-attention weights frames from multiple views for zero-shot action recognition","Zero-shot action recognition in untrimmed videos via composite attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3454,"prompt_tokens":978,"completion_tokens":2476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2372}},"tokens_in":594,"tokens_out":2476,"duration_ms":16318,"temperature":1.0,"reasoning_tokens":2372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:17:45.437445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Implement the self-attention equations of Section 3.1 with the stated shapes: a G×1 attention vector applied to an s×G feature matrix cannot produce an s×G output, so the model as written does not have a well-defined forward pass; a reader can check the released code or a clarified derivation for the missing operation and then verify whether it reproduces 98.7% on UCF101 and 89.7% on THUMOS14. Any implementation that silently patches the dimension mismatch would make the reported numbers unreproducible from the paper's own definitions.","supporting_citations":[{"cited_title":"In: AAAI (2019)","cited_arxiv_id":null,"evidence_quote":"supplies the two-stream graph convolutional zero-shot architecture whose attention module the paper replaces"},{"cited_title":"In: AAAI (2019)","cited_arxiv_id":null,"evidence_quote":"supplies the composite feature branch and the self-attention design for untrimmed action recognition that the paper extends"},{"cited_title":"I.: Learning transferable features with deep adaptation networks","cited_arxiv_id":null,"evidence_quote":"provides the generalized maximum mean discrepancy used to align trimmed and untrimmed streams in the transfer module"},{"cited_title":"In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2017) 12","cited_arxiv_id":null,"evidence_quote":"provides the pretrained two-stream baseline and pretraining context that the reported results are compared against"},{"cited_title":"International Journal of Computer Vision 123(3), 309-333(2017)","cited_arxiv_id":null,"evidence_quote":"defines the word-vector embedding approach and the data split the experiments follow"},{"cited_title":"In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , CVPR (2017)","cited_arxiv_id":null,"evidence_quote":"is the main untrimmed action recognition baseline the model claims to outperform"},{"cited_title":"In: ECCV, pp","cited_arxiv_id":null,"evidence_quote":"is the TSN baseline used for both trimmed and untrimmed comparisons"}],"review_version":1}