REVIEW 4 major objections 5 minor 37 references
Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Adversarial seed growing recovers full action boundaries from sparse CAS peaks and mines missed action regions, giving state-of-the-art weakly-supervised localization on THUMOS'14 and ActivityNet1.3.
desk verdict A legitimate application of seed-growing and erasing to temporal action localization with real high-IoU gains, but the 'adversarial' erasing mechanism is never isolated, so the causal story is weaker than the ablation suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the adversarial pair of the Seeded Sequence Growing (SSG) network and the erasing video-level classifier sharing one feature map. SSG owns a per-class temporal heatmap and a growing criterion $G(H_{c,t}, S_c, \theta_g)$ that labels a neighbor of an existing seed as class $c$ when the heatmap exceeds the growing threshold $\theta_g$ and the class is the argmax at that segment; this extends seeds toward action boundaries under the seeding loss $L_{seed}$. The classifier erases all segments whose heatmap exceeds the erasing threshold $\theta_a$, then aggregates the remaining features with Self-Adaptive Pooling (SAP)—a weighted sum whose attention weights come from the shared feature map itself—and is trained with cross-entropy $L_{class}$. The adversarial loop is the alternation: erasing forces the classifier to find class-relevant evidence in the leftover features, and the classifier's gradients, passed through the shared map, push the SSG to grow seeds into regions the classifier still needs, so the two losses alternately raise the difficulty for each other.
What would settle it
Train the full ASSG pipeline but replace the erased temporal segments with a randomly selected set of the same size as the SSG-activated regions; if the mAP gain over the SSG-only baseline is unchanged under this control, then the specific choice of which regions to erase is not what drives the classifier to recover missed actions, so the adversarial mechanism is not doing the claimed work.
Extended reading notes
Core claim
The central claim is that the two failure modes of CAS-based weakly-supervised localization—peak fragmentation of long actions and omission of trivial instances—can be removed by an adversarial loop rather than by stronger supervision. The SSG network initializes each action category with the sparse high-confidence regions of a CAS heatmap and then progressively labels neighboring segments according to a growing criterion that requires the segment's heatmap value to exceed a threshold and its class to be the maximum over all classes. In parallel, a self-adaptive classifier erases those SSG-activated high-confidence regions from the shared feature map and classifies the video from what remains, using a Self-Adaptive Pooling (SAP) aggregation that re-weights segments. Because the classifier must make the correct video-level prediction without the most discriminative features, it is pushed to attend to less salient or previously missed action segments, and the shared parameters mean this pressure also refines the heatmaps. The combination raises the average mAP over IoU 0.1–0.5 on THUMOS'14 from 34.2% with SSG alone to 43.5%, and reaches 25.4% at IoU 0.5 and 32.3% at IoU 0.5 on ActivityNet1.3, both above prior weakly-supervised methods.
Load-bearing premise
The assumption that carries the whole method is that after the SSG's high-confidence regions are erased, the remaining feature sequence still contains enough class-specific evidence about the non-salient or previously missed action instances for the classifier to recover them; if the erased features are information-free for those instances, the adversarial mining collapses to the seed-growing component alone.
Editorial extensions
If this is right
- Long and fragmented actions can be localized at high IoU without frame-level supervision, because boundary recovery is formulated as growing reliable seeds rather than as explicit boundary regression.
- Any CAS-based weakly-supervised detector can be augmented with the SSG and erasing classifier to improve its high-IoU performance, since the added module needs no new labels and the erasing classifier adds no parameters.
- The growing criterion's two thresholds ($\theta_g$ for expansion, $\theta_a$ for erasing) give a simple, dataset-robust control over the aggressiveness of growth and the amount of feature hiding, as the ablation shows performance changes little across threshold values.
- The adversarial training's main contribution is instance completeness and boundary refinement rather than coarse recall, since the gains concentrate at IoU 0.3–0.5 on THUMOS'14.
Reading between the lines
- The same adversarial erasing trick could be carried back to 2D weakly-supervised object localization: take CAM peaks as seeds, grow them with a seeded region-growing network, and erase the grown regions so a classifier is forced to attend to the rest of the object; the 1D formulation is a direct special case.
- The Self-Adaptive Pooling aggregation, which re-weights segments by their learned attention without extra parameters, may be useful for any weakly-supervised classification of long sequences (e.g., audio event detection) where background segments would otherwise dominate a global average pool.
- A testable next step is to use the SSG's grown heatmaps as pseudo ground-truth segment labels for training a fully-supervised detector, which could close most of the remaining gap to strong supervision without any manual segment annotation.
- Because the method inherits its seeds from a CAS model, its ceiling is tied to seed quality; running the adversarial loop on top of several diverse CAS initializations and fusing their heatmaps might improve robustness to seed failure.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Adversarial Seeded Sequence Growing (ASSG) for weakly-supervised temporal action localization. The method takes Class Activation Sequence (CAS) peaks as initial seeds, grows these seeds with a Seeded Sequence Growing (SSG) network, and adds a classifier branch that erases the high-confidence SSG-activated regions from the shared features and then predicts video-level classes via a self-adaptive pooling (SAP) aggregation. The two losses are alternated with the claim that this constitutes adversarial training. Experiments on THUMOS'14 and ActivityNet1.3 report large improvements over the CAS baseline (e.g., average mAP 0.1--0.5 from 24.4% to 34.2% with SSG and to 43.5% with ASSG in Table 2) and state-of-the-art or competitive results against other weakly supervised methods, including 25.4% mAP at IoU 0.5 on THUMOS'14 and 32.3% at IoU 0.5 on ActivityNet1.3.
Significance. If the causal mechanism claimed by the paper holds, ASSG would be a valuable advance: it combines seed growing with an erasing classifier to mine less discriminative action regions, and it reports consistent gains on two standard benchmarks. The paper has concrete strengths: ablations in Tables 1 and 2 show the contribution of the SAP pooling and of the overall two-module system, the evaluation is against external baselines, and the implementation details are sufficiently specific to be reproduced. However, the central claim that adversarial erasing is the driver of the improvement is not experimentally isolated, and the adversarial interpretation of the training procedure is not substantiated by analysis or ablation. These issues affect the validity of the paper's main explanatory narrative rather than merely its presentation.
major comments (4)
- [§3.3, Algorithm 1 lines 14–16, Table 2] The causal role of the erasing step is untested. The only architectural ablation compares SSG (CAS w/ SSG) with ASSG (CAS w/ SSG w/ classifier), so the 9.3% average-mAP gain (34.2% to 43.5% in Table 2) could come entirely from adding a second classification loss on shared features, independent of whether any features are erased. A convincing experiment must compare the full ASSG against a variant that trains the same classifier on unerased features, or against a random-erasing control, to show that erasing the SSG-activated regions is what drives the improvement.
- [§3.4, Algorithm 1, Section 1] The term 'adversarial' is used more strongly than the algorithm supports. Algorithm 1 simply alternates optimization of the seeding loss L_seed and the classification loss L_class; there is no minimax objective, gradient reversal, or equilibrium analysis. The authors should either provide evidence that the erasing step creates a harder classification distribution and that the alternation yields the claimed mining behavior, or revise the claims to describe the procedure as alternating optimization with feature erasure rather than true adversarial training.
- [§3.3, §4.3.4, Figure 3] No quantitative evidence is provided that the classifier actually recovers trivial or previously missed action instances. The only support is a single qualitative example in Figure 3 showing one missing instance emerging. The authors should measure, for example, the recall of ground-truth instances that are missed by the initial CAS seeds or by the SSG output, and compare it before and after the erasing-classifier branch is added. Without such a measure, the paper's central 'mining' claim remains a plausible but unverified interpretation of the aggregate mAP gain.
- [§4.2, §4.3.1] The thresholds θ_d, θ_a, and λ are selected on the same benchmark used for the final state-of-the-art comparison, with no held-out validation split and no error bars or multiple-run statistics. This is a robustness concern for the reported margins, especially the 9.3% gain in Table 2. The authors should report standard deviations over at least a few runs, or select hyperparameters on a validation portion and report the corresponding test results, to rule out selection bias.
minor comments (5)
- [Eq. (1)] The notation is inconsistent: G(H_{c,t}, S_c, θ_d) is written with a condition l ∈ N(S_c) but the heatmap index is t, and the neighbor variable l is not defined in terms of t. Please clarify the neighbor set and the indexing.
- [Eq. (3)] The definition of A_{c,t}(X) is ambiguous: the meaning of |f_{c,t}(X)| and the sum over i = 1 to |f_{c,t}(X)| of f^i_{c,t}(X) is unclear, and the relationship between f(·) and the earlier feature map F is not stated precisely.
- [Throughout] There are several typos and formatting artifacts: 'earsing' for 'erasing', 'Evalution' for 'Evaluation', 'Hanmmer' for 'Hammer', 'erasion' for 'erasure', and a stray glyph after the NMS sentence in Section 3.5. These should be corrected.
- [§3.3] The description of 'a pair of opposite ReLU activations' used to separate foreground and background features is too brief; the authors should explain how the positive and negative activation branches are constructed and how they are used to form the foreground features F.
- [Table 3 and §4.4] The text states that STAR without frequency annotations reports 44.0% average mAP, but the STAR row in Table 3 implies an average of about 47.0% over IoU 0.1–0.5. Please clarify which configuration each reported STAR number corresponds to.
Circularity Check
No significant circularity; external benchmark results support the central claim, with only minor self-citation from the authors' prior STAR work used as CAS initialization and baseline.
full rationale
The paper's central empirical claim, state-of-the-art weakly-supervised temporal action localization, is evaluated on external benchmarks (THUMOS'14 and ActivityNet1.3) against numerous external baselines; the reported mAP values are not derived by construction from the method's inputs. The seed-growing loop in Eqs. (1) and (4) is self-referential: the supervision set T_c is obtained by thresholding the network's own heatmap H, and L_seed then trains H to match T_c. This is a pseudo-labeling/self-training mechanism rather than a derivation in which the predicted quantity is algebraically identical to the input; the final detector is tested on held-out videos, so the state-of-the-art result is not manufactured by construction. The paper also relies on the authors' prior STAR work [28] for the default CAS initialization and as a reported baseline (Section 4.2: 'we realize the ST-GradCAM with the pre-defined parameters described in [28] as our default setting'; Table 2: 'Reported CAS [28]'). This is a minor self-citation, but it is not load-bearing: the paper also reports STPN-CAS w/ ASSG initialized from the external STPN method and shows improvement over STPN, and the main comparisons include many external weakly-supervised methods. No uniqueness theorem or ansatz is imported from the authors' prior work; the 'adversarial' component is simply an alternating optimization of two losses rather than a minimax argument. The main weakness is that the causal role of the erasing step is not isolated by an ablation, so the mechanism claim is under-supported; that is a missing experiment or correctness risk, not circularity.
Assumptions & free parameters
free parameters (3)
- growing threshold theta_d =
0.99 for foreground and background
- erasing threshold theta_a =
0.4
- modality ratio lambda =
0.3
assumptions (4)
- domain assumption Initial seeds from CAS peaks and saliency-detected shot changes are reliable enough to bootstrap growing.
- domain assumption The erased feature map still contains discriminative evidence for non-salient or missed action instances.
- domain assumption Neighboring unlabeled segments with high class probability and argmax class can be safely labeled as that class.
- ad hoc to paper The two-module training is a genuine adversarial process that improves localization.
Cite this review
Pith. "Pith review of Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization." pith.science (2026). https://pith.science/paper/OPLAVXHO
@misc{pith2026190802422,
author = {Pith},
title = {Pith review of: Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization},
year = {2026},
howpublished = {\url{https://pith.science/paper/OPLAVXHO}},
note = {Machine review of arXiv:1908.02422}
}
read the original abstract
Temporal action localization is an important yet challenging research topic due to its various applications. Since the frame-level or segment-level annotations of untrimmed videos require amounts of labor expenditure, studies on the weakly-supervised action detection have been springing up. However, most of existing frameworks rely on Class Activation Sequence (CAS) to localize actions by minimizing the video-level classification loss, which exploits the most discriminative parts of actions but ignores the minor regions. In this paper, we propose a novel weakly-supervised framework by adversarial learning of two modules for eliminating such demerits. Specifically, the first module is designed as a well-designed Seeded Sequence Growing (SSG) Network for progressively extending seed regions (namely the highly reliable regions initialized by a CAS-based framework) to their expected boundaries. The second module is a specific classifier for mining trivial or incomplete action regions, which is trained on the shared features after erasing the seeded regions activated by SSG. In this way, a whole network composed of these two modules can be trained in an adversarial manner. The goal of the adversary is to mine features that are difficult for the action classifier. That is, erasion from SSG will force the classifier to discover minor or even new action regions on the input feature sequence, and the classifier will drive the seeds to grow, alternately. At last, we could obtain the action locations and categories from the well-trained SSG and the classifier. Extensive experiments on two public benchmarks THUMOS'14 and ActivityNet1.3 demonstrate the impressive performance of our proposed method compared with the state-of-the-arts.
Figures
Reference graph
Works this paper leans on
-
[1]
Humam Alwassel, Fabian Caba Heilbron, and Bernard Ghanem. 2018. Action Search: Spotting Actions in Videos and Its Application to Temporal Action Local- ization. In ECCV. 251–266
work page 2018
-
[2]
Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. 2017. SST: Single-Stream Temporal Action Proposals. In CVPR. 6373– 6382
work page 2017
-
[3]
Joao Carreira and Andrew Zisserman. 2017. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In CVPR. 4724–4733
work page 2017
-
[4]
Yu Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. 2018. Rethinking the Faster R-CNN Architecture for Temporal Action Localization. In CVPR. 2933–2942
work page 2018
-
[5]
Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S. Davis, and Yan Qiu Chen
-
[6]
Jiyang Gao, Zhenheng Yang, and Ram Nevatia. 2017. Cascaded Boundary Regres- sion for Temporal Action Detection. CoRR abs/1705.01180 (2017)
arXiv 2017
-
[7]
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles
-
[8]
Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang
Show all 37 references
-
[9]
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolu- tional Architecture for Fast Feature Embedding. ACM MM, 675–678
2014
-
[10]
Jiang, J
Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Suk- thankar. 2014. THUMOS Challenge: Action Recognition with a Large Number of Classes. http://crcv.ucf.edu/THUMOS14/
2014
-
[11]
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset. CoRR abs/1705.06950 (2017)
2017 arXiv
-
[12]
Alexander Kolesnikov and Christoph H. Lampert. 2016. Seed, Expand and Con- strain: Three Principles for Weakly-Supervised Image Segmentation. In ECCV
2016
-
[13]
Tianwei Lin, Xu Zhao, and Zheng Shou. 2017. Single Shot Temporal Action Detection. In ACM MM. 988–996
2017
-
[14]
Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. 2018. BSN: Boundary Sensitive Network for Temporal Action Proposal Generation. In ECCV. 3–19
2018
-
[15]
Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. 2018. Weakly Su- pervised Action Localization by Sparse Temporal Pooling Network. In CVPR. 6752–6761
2018
-
[16]
Dan Oneata, Jakob J Verbeek, and Cordelia Schmid. 2014. Efficient Action Local- ization with Approximately Normalized Fisher Vectors. In CVPR. 2545–2552
2014
-
[17]
Sujoy Paul, Sourya Roy, Amit K Roy Chowdhury, and Amit K. 2018. W-TALC: Weakly-supervised Temporal Activity Localization and Classification. In ECCV. 563–579
2018
-
[18]
Alexander Richard and Juergen Gall. 2016. Temporal Action Detection Using a Statistical Language Model. In CVPR. 3131–3140
2016
-
[19]
Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih- Fu Chang. 2017. CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos. In CVPR. 5734–5743
2017
-
[20]
Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang
-
[21]
Zheng Shou, Dongang Wang, and Shih Fu Chang. 2016. Temporal Action Local- ization in Untrimmed Videos via Multi-stage CNNs. In CVPR. 1049–1058
2016
-
[22]
Gurkirt Singh and Fabio Cuzzolin. 2016. Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge. CoRR abs/1607.01979 (2016)
2016 arXiv
-
[23]
Krishna Kumar Singh and Jae Lee Yong. 2017. Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-Supervised Object and Action Localization. In ICCV. 3544–3553
2017
-
[24]
AutoLoc: Weakly-supervised Temporal Action Localization in Untrimmed Videos. In ECCV. 154–171
-
[25]
Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming Ming Cheng, Zhao Yao, and Shuicheng Yan. 2017. Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. In CVPR
2017
-
[26]
Yuanjun Xiong, Yue Zhao, Limin Wang, Dahua Lin, and Xiaoou Tang. 2017. A Pursuit of Temporal Accuracy in General Activity Detection.CoRR abs/1703.02716 (2017)
2017 arXiv
-
[27]
Huijuan Xu, Abir Das, and Kate Saenko. 2017. R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. In ICCV. 5783–5792
2017
-
[28]
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. 2017. UntrimmedNets for Weakly Supervised Action Recognition and Detection. In CVPR. 6402–6411
2017
-
[29]
Ke Yang, Peng Qiao, Dongsheng Li, Shaohe Lv, and Yong Dou. 2018. Exploring Temporal Preservation Networks for Precise Temporal Action Localization. In AAAI. 7477–7484
2018
-
[30]
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. 2017. Temporal Action Detection with Structured Segment Networks. In ICCV. 2933–2942
2017
-
[31]
Jia Xing Zhong, Nannan Li, Weijie Kong, Zhang Tao, and Li Ge. 2018. Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector. In ACM Multimedia Conference. 35–44
2018
-
[32]
Yunlu Xu, Chengwei Zhang, Zhanzhan Cheng, Jianwen Xie, Yi Niu, Shiliang Pu, and Fei Wu. 2019. Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection. AAAI
2019
-
[36]
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba
-
[2015]
ActivityNet: A large-scale video benchmark for human activity understand- ing. In CVPR. 961–970
-
[2016]
Learning Deep Features for Discriminative Localization. In CVPR. 2921– 2929
-
[2017]
Temporal Context Network for Activity Localization in Videos. In ICCV. 5727–5736
-
[2018]
Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing. In CVPR. 7014–7023
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.