Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Adversarial seed growing recovers full action boundaries from sparse CAS peaks and mines missed action regions, giving state-of-the-art weakly-supervised localization on THUMOS'14 and ActivityNet1.3.

desk verdict A legitimate application of seed-growing and erasing to temporal action localization with real high-IoU gains, but the 'adversarial' erasing mechanism is never isolated, so the causal story is weaker than the ablation suggests. read the letter →

arxiv 1908.02422 v1 pith:OPLAVXHO submitted 2019-08-07 cs.CV

classification cs.CV
keywords temporalactionlocalizationweaksupervisionadversariallearningseedgrowingclassactivationsequenceerasing-baseddetectionvideounderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on a known weakness of weakly-supervised temporal action localization: a Class Activation Sequence (CAS) trained on video-level labels fires only on the most discriminative fragments of an action, so long actions are detected as disconnected peaks and inconspicuous action instances are missed entirely. The proposed ASSG framework treats those CAS peaks as seeds and grows them outward with a Seeded Sequence Growing (SSG) network, while a second classifier, trained on the shared features after the seeded regions are erased, is forced to discover the less obvious action regions. The two modules are trained adversarially, so the erasing drives the classifier to find new evidence and the classifier's difficulty in turn pushes the seeds to grow further. If this works as claimed, high-quality temporal boundaries become recoverable from video-level class labels alone, and the paper reports state-of-the-art detection on THUMOS'14 and ActivityNet1.3, with the largest gains at the stricter IoU thresholds.

What carries the argument

The load-bearing object is the adversarial pair of the Seeded Sequence Growing (SSG) network and the erasing video-level classifier sharing one feature map. SSG owns a per-class temporal heatmap and a growing criterion $G(H_{c,t}, S_c, \theta_g)$ that labels a neighbor of an existing seed as class $c$ when the heatmap exceeds the growing threshold $\theta_g$ and the class is the argmax at that segment; this extends seeds toward action boundaries under the seeding loss $L_{seed}$. The classifier erases all segments whose heatmap exceeds the erasing threshold $\theta_a$, then aggregates the remaining features with Self-Adaptive Pooling (SAP)—a weighted sum whose attention weights come from the shared feature map itself—and is trained with cross-entropy $L_{class}$. The adversarial loop is the alternation: erasing forces the classifier to find class-relevant evidence in the leftover features, and the classifier's gradients, passed through the shared map, push the SSG to grow seeds into regions the classifier still needs, so the two losses alternately raise the difficulty for each other.

What would settle it

Train the full ASSG pipeline but replace the erased temporal segments with a randomly selected set of the same size as the SSG-activated regions; if the mAP gain over the SSG-only baseline is unchanged under this control, then the specific choice of which regions to erase is not what drives the classifier to recover missed actions, so the adversarial mechanism is not doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that the two failure modes of CAS-based weakly-supervised localization—peak fragmentation of long actions and omission of trivial instances—can be removed by an adversarial loop rather than by stronger supervision. The SSG network initializes each action category with the sparse high-confidence regions of a CAS heatmap and then progressively labels neighboring segments according to a growing criterion that requires the segment's heatmap value to exceed a threshold and its class to be the maximum over all classes. In parallel, a self-adaptive classifier erases those SSG-activated high-confidence regions from the shared feature map and classifies the video from what remains, using a Self-Adaptive Pooling (SAP) aggregation that re-weights segments. Because the classifier must make the correct video-level prediction without the most discriminative features, it is pushed to attend to less salient or previously missed action segments, and the shared parameters mean this pressure also refines the heatmaps. The combination raises the average mAP over IoU 0.1–0.5 on THUMOS'14 from 34.2% with SSG alone to 43.5%, and reaches 25.4% at IoU 0.5 and 32.3% at IoU 0.5 on ActivityNet1.3, both above prior weakly-supervised methods.

Load-bearing premise

The assumption that carries the whole method is that after the SSG's high-confidence regions are erased, the remaining feature sequence still contains enough class-specific evidence about the non-salient or previously missed action instances for the classifier to recover them; if the erased features are information-free for those instances, the adversarial mining collapses to the seed-growing component alone.

Editorial extensions

If this is right

  • Long and fragmented actions can be localized at high IoU without frame-level supervision, because boundary recovery is formulated as growing reliable seeds rather than as explicit boundary regression.
  • Any CAS-based weakly-supervised detector can be augmented with the SSG and erasing classifier to improve its high-IoU performance, since the added module needs no new labels and the erasing classifier adds no parameters.
  • The growing criterion's two thresholds ($\theta_g$ for expansion, $\theta_a$ for erasing) give a simple, dataset-robust control over the aggressiveness of growth and the amount of feature hiding, as the ablation shows performance changes little across threshold values.
  • The adversarial training's main contribution is instance completeness and boundary refinement rather than coarse recall, since the gains concentrate at IoU 0.3–0.5 on THUMOS'14.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same adversarial erasing trick could be carried back to 2D weakly-supervised object localization: take CAM peaks as seeds, grow them with a seeded region-growing network, and erase the grown regions so a classifier is forced to attend to the rest of the object; the 1D formulation is a direct special case.
  • The Self-Adaptive Pooling aggregation, which re-weights segments by their learned attention without extra parameters, may be useful for any weakly-supervised classification of long sequences (e.g., audio event detection) where background segments would otherwise dominate a global average pool.
  • A testable next step is to use the SSG's grown heatmaps as pseudo ground-truth segment labels for training a fully-supervised detector, which could close most of the remaining gap to strong supervision without any manual segment annotation.
  • Because the method inherits its seeds from a CAS model, its ceiling is tied to seed quality; running the adversarial loop on top of several diverse CAS initializations and fusing their heatmaps might improve robustness to seed failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Adversarial Seeded Sequence Growing (ASSG) for weakly-supervised temporal action localization. The method takes Class Activation Sequence (CAS) peaks as initial seeds, grows these seeds with a Seeded Sequence Growing (SSG) network, and adds a classifier branch that erases the high-confidence SSG-activated regions from the shared features and then predicts video-level classes via a self-adaptive pooling (SAP) aggregation. The two losses are alternated with the claim that this constitutes adversarial training. Experiments on THUMOS'14 and ActivityNet1.3 report large improvements over the CAS baseline (e.g., average mAP 0.1--0.5 from 24.4% to 34.2% with SSG and to 43.5% with ASSG in Table 2) and state-of-the-art or competitive results against other weakly supervised methods, including 25.4% mAP at IoU 0.5 on THUMOS'14 and 32.3% at IoU 0.5 on ActivityNet1.3.

Significance. If the causal mechanism claimed by the paper holds, ASSG would be a valuable advance: it combines seed growing with an erasing classifier to mine less discriminative action regions, and it reports consistent gains on two standard benchmarks. The paper has concrete strengths: ablations in Tables 1 and 2 show the contribution of the SAP pooling and of the overall two-module system, the evaluation is against external baselines, and the implementation details are sufficiently specific to be reproduced. However, the central claim that adversarial erasing is the driver of the improvement is not experimentally isolated, and the adversarial interpretation of the training procedure is not substantiated by analysis or ablation. These issues affect the validity of the paper's main explanatory narrative rather than merely its presentation.

major comments (4)
  1. [§3.3, Algorithm 1 lines 14–16, Table 2] The causal role of the erasing step is untested. The only architectural ablation compares SSG (CAS w/ SSG) with ASSG (CAS w/ SSG w/ classifier), so the 9.3% average-mAP gain (34.2% to 43.5% in Table 2) could come entirely from adding a second classification loss on shared features, independent of whether any features are erased. A convincing experiment must compare the full ASSG against a variant that trains the same classifier on unerased features, or against a random-erasing control, to show that erasing the SSG-activated regions is what drives the improvement.
  2. [§3.4, Algorithm 1, Section 1] The term 'adversarial' is used more strongly than the algorithm supports. Algorithm 1 simply alternates optimization of the seeding loss L_seed and the classification loss L_class; there is no minimax objective, gradient reversal, or equilibrium analysis. The authors should either provide evidence that the erasing step creates a harder classification distribution and that the alternation yields the claimed mining behavior, or revise the claims to describe the procedure as alternating optimization with feature erasure rather than true adversarial training.
  3. [§3.3, §4.3.4, Figure 3] No quantitative evidence is provided that the classifier actually recovers trivial or previously missed action instances. The only support is a single qualitative example in Figure 3 showing one missing instance emerging. The authors should measure, for example, the recall of ground-truth instances that are missed by the initial CAS seeds or by the SSG output, and compare it before and after the erasing-classifier branch is added. Without such a measure, the paper's central 'mining' claim remains a plausible but unverified interpretation of the aggregate mAP gain.
  4. [§4.2, §4.3.1] The thresholds θ_d, θ_a, and λ are selected on the same benchmark used for the final state-of-the-art comparison, with no held-out validation split and no error bars or multiple-run statistics. This is a robustness concern for the reported margins, especially the 9.3% gain in Table 2. The authors should report standard deviations over at least a few runs, or select hyperparameters on a validation portion and report the corresponding test results, to rule out selection bias.
minor comments (5)
  1. [Eq. (1)] The notation is inconsistent: G(H_{c,t}, S_c, θ_d) is written with a condition l ∈ N(S_c) but the heatmap index is t, and the neighbor variable l is not defined in terms of t. Please clarify the neighbor set and the indexing.
  2. [Eq. (3)] The definition of A_{c,t}(X) is ambiguous: the meaning of |f_{c,t}(X)| and the sum over i = 1 to |f_{c,t}(X)| of f^i_{c,t}(X) is unclear, and the relationship between f(·) and the earlier feature map F is not stated precisely.
  3. [Throughout] There are several typos and formatting artifacts: 'earsing' for 'erasing', 'Evalution' for 'Evaluation', 'Hanmmer' for 'Hammer', 'erasion' for 'erasure', and a stray glyph after the NMS sentence in Section 3.5. These should be corrected.
  4. [§3.3] The description of 'a pair of opposite ReLU activations' used to separate foreground and background features is too brief; the authors should explain how the positive and negative activation branches are constructed and how they are used to form the foreground features F.
  5. [Table 3 and §4.4] The text states that STAR without frequency annotations reports 44.0% average mAP, but the STAR row in Table 3 implies an average of about 47.0% over IoU 0.1–0.5. Please clarify which configuration each reported STAR number corresponds to.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; external benchmark results support the central claim, with only minor self-citation from the authors' prior STAR work used as CAS initialization and baseline.

full rationale

The paper's central empirical claim, state-of-the-art weakly-supervised temporal action localization, is evaluated on external benchmarks (THUMOS'14 and ActivityNet1.3) against numerous external baselines; the reported mAP values are not derived by construction from the method's inputs. The seed-growing loop in Eqs. (1) and (4) is self-referential: the supervision set T_c is obtained by thresholding the network's own heatmap H, and L_seed then trains H to match T_c. This is a pseudo-labeling/self-training mechanism rather than a derivation in which the predicted quantity is algebraically identical to the input; the final detector is tested on held-out videos, so the state-of-the-art result is not manufactured by construction. The paper also relies on the authors' prior STAR work [28] for the default CAS initialization and as a reported baseline (Section 4.2: 'we realize the ST-GradCAM with the pre-defined parameters described in [28] as our default setting'; Table 2: 'Reported CAS [28]'). This is a minor self-citation, but it is not load-bearing: the paper also reports STPN-CAS w/ ASSG initialized from the external STPN method and shows improvement over STPN, and the main comparisons include many external weakly-supervised methods. No uniqueness theorem or ansatz is imported from the authors' prior work; the 'adversarial' component is simply an alternating optimization of two losses rather than a minimax argument. The main weakness is that the causal role of the erasing step is not isolated by an ablation, so the mechanism claim is under-supported; that is a missing experiment or correctness risk, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of initial seeds, the presence of recoverable signal in erased features, and the effectiveness of the growing rule; these are domain assumptions about the data and feature space, not derived or independently verified.

free parameters (3)
  • growing threshold theta_d = 0.99 for foreground and background
    Set in Section 4.2; ablation in Section 4.3.1 shows small influence. Used to decide whether to label a segment in seed growing (Eq. 1).
  • erasing threshold theta_a = 0.4
    Set in Section 4.2; ablation in Section 4.3.1 shows performance peaks at 0.4. Used to erase high-activation regions in Algorithm 1 line 14.
  • modality ratio lambda = 0.3
    Set empirically in Section 4.2 for fusing RGB and flow heatmaps (Eq. 6).
assumptions (4)
  • domain assumption Initial seeds from CAS peaks and saliency-detected shot changes are reliable enough to bootstrap growing.
    Section 3.2.1; if seeds are noisy or miss complete instances, the growing rule can propagate errors. No quantitative analysis of seed quality is provided.
  • domain assumption The erased feature map still contains discriminative evidence for non-salient or missed action instances.
    Section 3.3 and Algorithm 1 lines 14-16; the classifier's ability to mine new regions depends on this premise.
  • domain assumption Neighboring unlabeled segments with high class probability and argmax class can be safely labeled as that class.
    Growing criterion Eq. (1); assumes local smoothness of action labels.
  • ad hoc to paper The two-module training is a genuine adversarial process that improves localization.
    Section 3.4.3 and Algorithm 1; the paper uses the term adversarial but no adversarial objective or game-theoretic formulation is provided. The mechanism is alternating erasing and classification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization." pith.science (2026). https://pith.science/paper/OPLAVXHO

@misc{pith2026190802422,
  author       = {Pith},
  title        = {Pith review of: Adversarial Seeded Sequence Growing for Weakly-Supervised Temporal Action Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPLAVXHO}},
  note         = {Machine review of arXiv:1908.02422}
}
read the original abstract

Temporal action localization is an important yet challenging research topic due to its various applications. Since the frame-level or segment-level annotations of untrimmed videos require amounts of labor expenditure, studies on the weakly-supervised action detection have been springing up. However, most of existing frameworks rely on Class Activation Sequence (CAS) to localize actions by minimizing the video-level classification loss, which exploits the most discriminative parts of actions but ignores the minor regions. In this paper, we propose a novel weakly-supervised framework by adversarial learning of two modules for eliminating such demerits. Specifically, the first module is designed as a well-designed Seeded Sequence Growing (SSG) Network for progressively extending seed regions (namely the highly reliable regions initialized by a CAS-based framework) to their expected boundaries. The second module is a specific classifier for mining trivial or incomplete action regions, which is trained on the shared features after erasing the seeded regions activated by SSG. In this way, a whole network composed of these two modules can be trained in an adversarial manner. The goal of the adversary is to mine features that are difficult for the action classifier. That is, erasion from SSG will force the classifier to discover minor or even new action regions on the input feature sequence, and the classifier will drive the seeds to grow, alternately. At last, we could obtain the action locations and categories from the well-trained SSG and the classifier. Extensive experiments on two public benchmarks THUMOS'14 and ActivityNet1.3 demonstrate the impressive performance of our proposed method compared with the state-of-the-arts.

Figures

Figures reproduced from arXiv: 1908.02422 by the authors.

Figure 1
Figure 1. Seed-Grow Mechanism for Action Localization. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The proposed ASSG architecture. (a) Encoded segmental features from video inputs. (b) The SSG module, with CAS [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of action localization by ASSG network. Temporal confidence distribution (the predicted heatmap in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance with different thresholds values. Fig [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 30 canonical work pages

  1. [1]

    Humam Alwassel, Fabian Caba Heilbron, and Bernard Ghanem. 2018. Action Search: Spotting Actions in Videos and Its Application to Temporal Action Local- ization. In ECCV. 251–266

  2. [2]

    Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. 2017. SST: Single-Stream Temporal Action Proposals. In CVPR. 6373– 6382

  3. [3]

    Joao Carreira and Andrew Zisserman. 2017. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In CVPR. 4724–4733

  4. [4]

    Yu Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. 2018. Rethinking the Faster R-CNN Architecture for Temporal Action Localization. In CVPR. 2933–2942

  5. [5]

    Davis, and Yan Qiu Chen

    Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S. Davis, and Yan Qiu Chen

  6. [6]

    Jiyang Gao, Zhenheng Yang, and Ram Nevatia. 2017. Cascaded Boundary Regres- sion for Temporal Action Detection. CoRR abs/1705.01180 (2017)

  7. [7]

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles

  8. [8]

    Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang

Show all 37 references
  1. [9]

    Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolu- tional Architecture for Fast Feature Embedding. ACM MM, 675–678

  2. [10]

    Jiang, J

    Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Suk- thankar. 2014. THUMOS Challenge: Action Recognition with a Large Number of Classes. http://crcv.ucf.edu/THUMOS14/

  3. [11]

    Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset. CoRR abs/1705.06950 (2017)

  4. [12]

    Alexander Kolesnikov and Christoph H. Lampert. 2016. Seed, Expand and Con- strain: Three Principles for Weakly-Supervised Image Segmentation. In ECCV

  5. [13]

    Tianwei Lin, Xu Zhao, and Zheng Shou. 2017. Single Shot Temporal Action Detection. In ACM MM. 988–996

  6. [14]

    Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. 2018. BSN: Boundary Sensitive Network for Temporal Action Proposal Generation. In ECCV. 3–19

  7. [15]

    Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. 2018. Weakly Su- pervised Action Localization by Sparse Temporal Pooling Network. In CVPR. 6752–6761

  8. [16]

    Dan Oneata, Jakob J Verbeek, and Cordelia Schmid. 2014. Efficient Action Local- ization with Approximately Normalized Fisher Vectors. In CVPR. 2545–2552

  9. [17]

    Sujoy Paul, Sourya Roy, Amit K Roy Chowdhury, and Amit K. 2018. W-TALC: Weakly-supervised Temporal Activity Localization and Classification. In ECCV. 563–579

  10. [18]

    Alexander Richard and Juergen Gall. 2016. Temporal Action Detection Using a Statistical Language Model. In CVPR. 3131–3140

  11. [19]

    Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih- Fu Chang. 2017. CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos. In CVPR. 5734–5743

  12. [20]

    Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang

  13. [21]

    Zheng Shou, Dongang Wang, and Shih Fu Chang. 2016. Temporal Action Local- ization in Untrimmed Videos via Multi-stage CNNs. In CVPR. 1049–1058

  14. [22]

    Gurkirt Singh and Fabio Cuzzolin. 2016. Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge. CoRR abs/1607.01979 (2016)

  15. [23]

    Krishna Kumar Singh and Jae Lee Yong. 2017. Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-Supervised Object and Action Localization. In ICCV. 3544–3553

  16. [24]

    AutoLoc: Weakly-supervised Temporal Action Localization in Untrimmed Videos. In ECCV. 154–171

  17. [25]

    Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming Ming Cheng, Zhao Yao, and Shuicheng Yan. 2017. Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. In CVPR

  18. [26]

    Yuanjun Xiong, Yue Zhao, Limin Wang, Dahua Lin, and Xiaoou Tang. 2017. A Pursuit of Temporal Accuracy in General Activity Detection.CoRR abs/1703.02716 (2017)

  19. [27]

    Huijuan Xu, Abir Das, and Kate Saenko. 2017. R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. In ICCV. 5783–5792

  20. [28]

    Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. 2017. UntrimmedNets for Weakly Supervised Action Recognition and Detection. In CVPR. 6402–6411

  21. [29]

    Ke Yang, Peng Qiao, Dongsheng Li, Shaohe Lv, and Yong Dou. 2018. Exploring Temporal Preservation Networks for Precise Temporal Action Localization. In AAAI. 7477–7484

  22. [30]

    Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. 2017. Temporal Action Detection with Structured Segment Networks. In ICCV. 2933–2942

  23. [31]

    Jia Xing Zhong, Nannan Li, Weijie Kong, Zhang Tao, and Li Ge. 2018. Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector. In ACM Multimedia Conference. 35–44

  24. [32]

    Yunlu Xu, Chengwei Zhang, Zhanzhan Cheng, Jianwen Xie, Yi Niu, Shiliang Pu, and Fei Wu. 2019. Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection. AAAI

  25. [36]

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba

  26. [2015]

    ActivityNet: A large-scale video benchmark for human activity understand- ing. In CVPR. 961–970

  27. [2016]

    Learning Deep Features for Discriminative Localization. In CVPR. 2921– 2929

  28. [2017]

    Temporal Context Network for Activity Localization in Videos. In ICCV. 5727–5736

  29. [2018]

    Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing. In CVPR. 7014–7023

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.