REVIEW 3 major objections 5 minor 34 references
3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Video-level labels plus action counts, trained with classification, center, and counting losses, sharpen temporal action localization to beat prior weak supervision by 4.6% mAP on THUMOS14 and 3.7% on ActivityNet 1.2.
desk verdict A solid empirical paper whose headline counting mechanism is mathematically misdescribed: Eq. 9 computes a softmax-weighted average, not an action count. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the temporal class activation map, a per-segment vector of class scores learned from video-level labels. Three mechanisms shape it: a top-k classification loss for video-level class separability; a multi-label center loss with class-specific centers $c_j$ and attention-weighted feature aggregation $f^a_i(j)=\sum_t a^a_i(t,j)\,x^a_i(t)/\sum_t a^a_i(t,j)$ pulling same-class features together; and a counting loss, $L_{\text{count}}$, that compares attention-weighted T-CAM sums to ground-truth counts, using relative error for present classes and absolute error for absent classes. The counting loss's inverse-count weighting makes low-count videos—exactly the cases where adjacent instances are easiest to merge—dominate the gradient.
What would settle it
Shuffle the ground-truth count labels across THUMOS14 training videos while keeping the same multiset of counts. If mAP at IoU=0.5 remains near 26.6, the specific count values are not carrying the signal. A complementary check is to correlate predicted counts $m_i(j)$ with true counts on held-out videos: near-zero correlation with unchanged localization would show the term acts as a generic regularizer rather than an actual count estimator.
Extended reading notes
Core claim
The central discovery is that jointly optimizing classification, center, and counting losses yields a temporal class activation map (T-CAM) whose high-scoring segments align with individual action instances rather than with the whole action-containing region. The classification term provides video-level category separability; the adapted multi-label center loss clusters features per class using attention-weighted aggregation over high-attention segments; the counting term compares the attention-weighted T-CAM sum $m_i(j)=\sum_t a^F_i(t,j)\,C^F_i(t,j)$ with the ground-truth count $n_i(j)$ and penalizes their relative difference. Together these losses push the T-CAM to be near zero in background and between-instance gaps. The paper reports 26.6 mAP at IoU=0.5 on THUMOS14 (up from 22.0 for [16]) and 21.7 mean mAP on ActivityNet 1.2 (up from 18.0 for [16]), with the center loss alone already beating prior weakly supervised methods.
Load-bearing premise
The method assumes the predicted quantity from the score curve really is the number of action instances, not just a weighted average of activation strengths; if that identification fails, the extra loss term may still regularize training but it is not performing the counting the paper describes.
Editorial extensions
If this is right
- Video-level category labels plus count labels can replace frame-level boundary annotations for training localizers, lowering annotation cost and reducing boundary subjectivity.
- The multi-label center loss alone—without count supervision—already surpasses prior weakly supervised localization, so the technique transfers to settings where counts are unavailable.
- The counting loss with relative-error weighting concentrates learning on low-count videos, the regime where instance merging most hurts average precision.
- Penalizing absent categories toward zero predicted count suppresses background and false detections; the paper's ablation attributes 1.5 mAP to this term.
- Joint training also improves action classification, reaching 86.9 mAP on THUMOS14, so the same model serves both recognition and localization.
Reading between the lines
- If the count term is mostly a regularizer, a cheap way to test this is to replace true counts with a learned count-prediction head trained on a small labeled subset; most of the gain persisting would mean per-video counts do not need to be manually annotated.
- The attention-weighted center loss is not tied to action localization; it could be applied to other multi-label temporal tasks such as sound event detection or gesture spotting, where category presence is known but instance boundaries are costly.
- On ActivityNet 1.2 the count term adds much less than on THUMOS14, consistent with the paper's observation of fewer multi-instance videos; this predicts that datasets with many repeated actions will show the largest benefit from count supervision.
- Because the counting module itself has no learnable parameters, a natural ablation is to replace it with a penalty on the temporal derivative of the T-CAM; if the two behave similarly, the gain is smoothness pressure rather than information about instance number.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes 3C-Net for weakly-supervised temporal action localization using video-level class labels and action instance counts. The architecture is a two-stream I3D-based network with three losses: a classification loss on top-k pooled T-CAMs, a multi-label center loss with attention-based feature aggregation, and a counting loss. Experiments on THUMOS14 and ActivityNet 1.2 report state-of-the-art results, with ablations showing progressive improvements from each term. The main claimed novelty is the counting loss, which is intended to separate adjacent action instances by matching a predicted count to the ground-truth count.
Significance. If the counting mechanism were correctly formulated, the idea of using instance counts as weak supervision for temporal action localization would be a useful contribution, especially since count labels are cheap relative to frame-level annotations. The multi-label adaptation of the center loss is also a sensible extension. The paper provides source code, thorough ablations, and comparisons on two benchmarks. However, the central counting loss as defined is mathematically a weighted average rather than a count, so the claimed mechanistic explanation is not supported; the empirical gains may still hold but require a corrected formulation and a re-evaluation of the underlying claim.
major comments (3)
- [Section 3.4, Eq. (9)] Equation (9) defines m_i(j) = sum_t a^F_i(t,j) C^F_i(t,j). Since a^F_i(t,j) is a temporal softmax over t, sum_t a^F_i(t,j) = 1, so m_i(j) is a weighted average of the T-CAM values over time. A weighted average cannot represent an instance count: one can rescale all C^F_i(t,j) and obtain any desired average, so a single-instance video and a fifteen-instance video can have identical m_i(j). Consequently, the counting loss in Eq. (10) does not force the activation mass to match the count; it forces a normalized average toward the integer label. This invalidates the claim in Section 1 that 'the prediction scores sum up to a positive value within action instances and zero otherwise' and the explanation that the loss delineates adjacent instances. The observed improvement in Table 4 (26.6 vs. 24.6 mAP) may be a regularization effect, but it is not evidence for count-based delineation.
- [Section 3.4, density-map claim] The paper states that the attention-weighted T-CAM is 'equivalent to a density map' and that its summation yields the predicted count. This analogy with object counting [6] is unsupported: in density-based counting, the density map is a spatial map whose integral equals the count, whereas here C^F is a class score produced by a classifier and the softmax weights are normalized to sum to one. Therefore the quantity in Eq. (9) is not an integral of a density. Furthermore, the claim that 'temporal attention weighting ignores the background video segments' is not implemented, because the softmax attention assigns nonzero (albeit small) weights to all segments and no threshold is applied in the counting module. The authors should either redefine m_i as a true sum over T-CAM values (with hard attention masking) or remove the density-map/count interpretation.
- [Section 4.3 and Table 4] The counting loss is computed as a per-video, per-class scalar after summing over time, so it cannot by itself separate adjacent instances; it provides only a global constraint on the T-CAM. The paper does not report any direct evidence that the counting loss improves instance-level segmentation, such as the number of detected instances versus the ground-truth count, or that it is responsible for the qualitative delineation shown in Fig. 1. The ablation in Table 4 isolates the loss, but it does not test the proposed mechanism. Without such evidence, the claim that the counting loss 'delineates adjacent action instances' is not established.
minor comments (5)
- [Section 3.4] There is a typo: 'disrcriminative' should be 'discriminative'.
- [Figure 1 caption] The caption reads 'PoleVaultaction'; it should be 'PoleVault action'.
- [Equation (7)] The denominator '1 + sum_i y_i(j)' is unusual; standard center updates divide by the number of samples in the class. Please clarify whether this is intentional to avoid division by zero, and explain the impact on the center update.
- [Equation (10)] The notation n_i(j) is introduced as the ground-truth count, but earlier y_i is a binary multi-hot label. Please define n_i explicitly and state its relationship to y_i.
- [Section 4.1] The description of the detection threshold η for ActivityNet 1.2 is terse; please state explicitly that η is set to 0 for that dataset and how the detection algorithm then generates segments from the T-CAM.
Circularity Check
No circularity: the count and center losses are supervised by external video-level labels and evaluated on held-out benchmarks, so no claimed result reduces to its own input.
full rationale
The derivation is self-contained and non-circular. The classification loss uses video-level multi-hot category labels; the center loss learns class centers from features aggregated by attention and is penalized against those external labels; the counting loss compares the model's predicted count m_i(j) (Eq. 9) with ground-truth counts n_i(j) (Eq. 10) that are generated from temporal segment annotations on the training split. Neither the predicted count nor the localization output is fitted to the evaluation target: localization is measured by mAP on held-out THUMOS14 test and ActivityNet 1.2 validation videos. The only self-citation is reference [6], used to support the density-map analogy for attention-weighted activations; that prior object-counting work is independently published and evaluated, so it does not smuggle in the present paper's conclusion. The reader's Eq. 9 concern is a legitimate mathematical correctness issue: because the temporal softmax in Eq. 4 sums to one over time, m_i(j) is a weighted average of T-CAM values rather than an instance count. That, however, is not a circularity: the prediction is still computed from the model and supervised by external count labels, and the claimed localization gains are empirically evaluated rather than derived from the count-loss definition.
Assumptions & free parameters
free parameters (4)
- alpha (center loss weight) =
1e-3
- beta (counting loss weight) =
1 (THUMOS14), 0.1 (ActivityNet 1.2)
- lambda (negative-count loss weight) =
1e-3
- top-k pooling ratio =
k = ceil(si/8)
assumptions (4)
- domain assumption Fixed pretrained I3D features from Kinetics are a sufficient representation for action localization.
- ad hoc to paper The attention-weighted sum in Eq. 9 yields an action-instance count.
- domain assumption Ground-truth action counts can be annotated independently of temporal boundaries.
- domain assumption Thresholding the T-CAM produces valid action instance detections.
Cite this review
Pith. "Pith review of 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization." pith.science (2026). https://pith.science/paper/6ULEADP3
@misc{pith2026190808216,
author = {Pith},
title = {Pith review of: 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization},
year = {2026},
howpublished = {\url{https://pith.science/paper/6ULEADP3}},
note = {Machine review of arXiv:1908.08216}
}
read the original abstract
Temporal action localization is a challenging computer vision problem with numerous real-world applications. Most existing methods require laborious frame-level supervision to train action localization models. In this work, we propose a framework, called 3C-Net, which only requires video-level supervision (weak supervision) in the form of action category labels and the corresponding count. We introduce a novel formulation to learn discriminative action features with enhanced localization capabilities. Our joint formulation has three terms: a classification term to ensure the separability of learned action features, an adapted multi-label center loss term to enhance the action feature discriminability and a counting loss term to delineate adjacent action sequences, leading to improved localization. Comprehensive experiments are performed on two challenging benchmarks: THUMOS14 and ActivityNet 1.2. Our approach sets a new state-of-the-art for weakly-supervised temporal action localization on both datasets. On the THUMOS14 dataset, the proposed method achieves an absolute gain of 4.6% in terms of mean average precision (mAP), compared to the state-of-the-art. Source code is available at https://github.com/naraysa/3c-net.
Figures
Reference graph
Works this paper leans on
-
[6]
Object counting and instance segmentation with image-level supervision
Hisham Cholakkal, Guolei Sun, Fahad Shahbaz Khan, and Ling Shao. Object counting and instance segmentation with image-level supervision. In CVPR, 2019. 2, 5
work page 2019
-
[1]
Finding actors and actions in movies
Piotr Bojanowski, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Finding actors and actions in movies. In ICCV, 2013. 1, 2
work page 2013
-
[2]
Weakly su- pervised action labeling in videos under ordering constraints
Piotr Bojanowski, R ´emi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly su- pervised action labeling in videos under ordering constraints. In ECCV, 2014. 2
work page 2014
-
[3]
Activitynet: A large-scale video benchmark for human activity understanding
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR,
-
[4]
Quo vadis, action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR,
-
[5]
Re- thinking the faster r-cnn architecture for temporal action lo- calization
Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Sey- bold, David A Ross, Jia Deng, and Rahul Sukthankar. Re- thinking the faster r-cnn architecture for temporal action lo- calization. In CVPR, 2018. 1, 6
work page 2018
-
[7]
Automatic annotation of human actions in video
Olivier Duchenne, Ivan Laptev, Josef Sivic, Francis Bach, and Jean Ponce. Automatic annotation of human actions in video. In ICCV, 2009. 2
work page 2009
-
[8]
C-wsl: Count-guided weakly supervised lo- calization
Mingfei Gao, Ang Li, Ruichi Yu, Vlad I Morariu, and Larry S Davis. C-wsl: Count-guided weakly supervised lo- calization. In ECCV, 2018. 2
work page 2018
Show all 34 references
-
[9]
The thu- mos challenge on action recognition for videos in the wild
Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thu- mos challenge on action recognition for videos in the wild. CVIU, 2017. 2, 6
2017
-
[10]
What do 15,000 object categories tell us about classifying and lo- calizing actions? In CVPR, 2015
Mihir Jain, Jan C Van Gemert, and Cees GM Snoek. What do 15,000 object categories tell us about classifying and lo- calizing actions? In CVPR, 2015. 7
2015
-
[11]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
-
[12]
Learning realistic human actions from movies
Ivan Laptev, Marcin Marszałek, Cordelia Schmid, and Ben- jamin Rozenfeld. Learning realistic human actions from movies. In CVPR, 2008. 1, 2
2008
-
[13]
Spot on: Action localization from pointly-supervised proposals
Pascal Mettes, Jan C Van Gemert, and Cees GM Snoek. Spot on: Action localization from pointly-supervised proposals. In ECCV, 2016. 1, 2
2016
-
[14]
Weakly supervised action localization by sparse temporal pooling network
Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. Weakly supervised action localization by sparse temporal pooling network. In CVPR, 2018. 1, 2, 3, 6
2018
-
[15]
The lear submission at thumos 2014
Dan Oneata, Jakob Verbeek, and Cordelia Schmid. The lear submission at thumos 2014. 2013. 1, 6
2014
-
[16]
W- talc: Weakly-supervised temporal activity localization and classification
Sujoy Paul, Sourya Roy, and Amit K Roy-Chowdhury. W- talc: Weakly-supervised temporal activity localization and classification. In ECCV, 2018. 1, 3, 5, 6, 7
2018
-
[17]
Weakly supervised action learning with rnn based fine-to-coarse modeling
Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In CVPR, 2017. 2
2017
-
[18]
Modeling the temporal ex- tent of actions
Scott Satkin and Martial Hebert. Modeling the temporal ex- tent of actions. In ECCV, 2010. 1
2010
-
[19]
Smoothing and differentiation of data by simplified least squares procedures
Abraham Savitzky and Marcel JE Golay. Smoothing and differentiation of data by simplified least squares procedures. Analytical chemistry, 36(8):1627–1639, 1964. 6
1964
-
[20]
Action snippets: How many frames does human action recognition require? In CVPR, 2008
Konrad Schindler and Luc Van Gool. Action snippets: How many frames does human action recognition require? In CVPR, 2008. 1
2008
-
[21]
Cdc: Convolutional-de- convolutional networks for precise temporal action localiza- tion in untrimmed videos
Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. Cdc: Convolutional-de- convolutional networks for precise temporal action localiza- tion in untrimmed videos. In CVPR, 2017. 1, 6
2017
-
[22]
Autoloc: weakly-supervised temporal action localization in untrimmed videos
Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang. Autoloc: weakly-supervised temporal action localization in untrimmed videos. In ECCV, 2018. 6, 7
2018
-
[23]
Temporal action localization in untrimmed videos via multi-stage cnns
Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In CVPR, 2016. 1, 6
2016
-
[24]
Two-stream con- volutional networks for action recognition in videos
Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In NIPS, 2014. 7
2014
-
[25]
Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization
Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In ICCV, 2017. 1, 2
2017
-
[26]
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015. 7
2015
-
[27]
Action recognition with improved trajectories
Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, 2013. 7
2013
-
[28]
Untrimmednets for weakly supervised action recognition and detection
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In CVPR, 2017. 1, 2, 3, 5, 6, 7
2017
-
[29]
Temporal segment networks: Towards good practices for deep action recogni- tion
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, 2016. 3, 6, 7
2016
-
[30]
A discriminative feature learning approach for deep face recog- nition
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recog- nition. In ECCV, 2016. 1, 3, 4, 5
2016
-
[31]
R-c3d: Region convolutional 3d network for temporal activity detection
Huijuan Xu, Abir Das, and Kate Saenko. R-c3d: Region convolutional 3d network for temporal activity detection. In ICCV, 2017. 1, 6
2017
-
[32]
A du- ality based approach for realtime tv-l 1 optical flow
Christopher Zach, Thomas Pock, and Horst Bischof. A du- ality based approach for realtime tv-l 1 optical flow. In Joint pattern recognition symposium, 2007. 6
2007
-
[33]
Temporal action detection with structured segment networks
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xi- aoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, 2017. 1, 6, 7
2017
-
[34]
Learning deep features for discrimi- native localization
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discrimi- native localization. In CVPR, 2016. 3 9
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.