Pith. sign in

REVIEW 3 major objections 5 minor 34 references

3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Video-level labels plus action counts, trained with classification, center, and counting losses, sharpen temporal action localization to beat prior weak supervision by 4.6% mAP on THUMOS14 and 3.7% on ActivityNet 1.2.

desk verdict A solid empirical paper whose headline counting mechanism is mathematically misdescribed: Eq. 9 computes a softmax-weighted average, not an action count. read the letter →

arxiv 1908.08216 v2 pith:6ULEADP3 submitted 2019-08-22 cs.CV

classification cs.CV
keywords weaklysupervisedtemporalactionlocalizationcountcenterlossclassactivationmapsvideo-levelsupervisionmulti-labelclassificationTHUMOS14ActivityNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Temporal action localization—deciding when an action happens in an untrimmed video—usually needs frame-level boundary annotations, which are expensive and subjective. This paper claims that video-level category labels plus a simple count of how many times each action appears are enough to train a strong localizer, and that adding a counting loss specifically helps separate adjacent action instances that otherwise merge into one detection. The proposed 3C-Net combines classification, multi-label center, and counting losses on a two-stream I3D feature pipeline. On THUMOS14 it raises mAP at IoU=0.5 from 19.1 (classification only) to 26.6, an absolute gain of 4.6% over the prior weakly supervised state of the art, and on ActivityNet 1.2 it gains 3.7 mean mAP. If the claim holds, annotating videos by category and count is a much cheaper route to usable localization than drawing temporal boxes.

What carries the argument

The load-bearing object is the temporal class activation map, a per-segment vector of class scores learned from video-level labels. Three mechanisms shape it: a top-k classification loss for video-level class separability; a multi-label center loss with class-specific centers $c_j$ and attention-weighted feature aggregation $f^a_i(j)=\sum_t a^a_i(t,j)\,x^a_i(t)/\sum_t a^a_i(t,j)$ pulling same-class features together; and a counting loss, $L_{\text{count}}$, that compares attention-weighted T-CAM sums to ground-truth counts, using relative error for present classes and absolute error for absent classes. The counting loss's inverse-count weighting makes low-count videos—exactly the cases where adjacent instances are easiest to merge—dominate the gradient.

What would settle it

Shuffle the ground-truth count labels across THUMOS14 training videos while keeping the same multiset of counts. If mAP at IoU=0.5 remains near 26.6, the specific count values are not carrying the signal. A complementary check is to correlate predicted counts $m_i(j)$ with true counts on held-out videos: near-zero correlation with unchanged localization would show the term acts as a generic regularizer rather than an actual count estimator.

Watch

Extended reading notes

Core claim

The central discovery is that jointly optimizing classification, center, and counting losses yields a temporal class activation map (T-CAM) whose high-scoring segments align with individual action instances rather than with the whole action-containing region. The classification term provides video-level category separability; the adapted multi-label center loss clusters features per class using attention-weighted aggregation over high-attention segments; the counting term compares the attention-weighted T-CAM sum $m_i(j)=\sum_t a^F_i(t,j)\,C^F_i(t,j)$ with the ground-truth count $n_i(j)$ and penalizes their relative difference. Together these losses push the T-CAM to be near zero in background and between-instance gaps. The paper reports 26.6 mAP at IoU=0.5 on THUMOS14 (up from 22.0 for [16]) and 21.7 mean mAP on ActivityNet 1.2 (up from 18.0 for [16]), with the center loss alone already beating prior weakly supervised methods.

Load-bearing premise

The method assumes the predicted quantity from the score curve really is the number of action instances, not just a weighted average of activation strengths; if that identification fails, the extra loss term may still regularize training but it is not performing the counting the paper describes.

Editorial extensions

If this is right

  • Video-level category labels plus count labels can replace frame-level boundary annotations for training localizers, lowering annotation cost and reducing boundary subjectivity.
  • The multi-label center loss alone—without count supervision—already surpasses prior weakly supervised localization, so the technique transfers to settings where counts are unavailable.
  • The counting loss with relative-error weighting concentrates learning on low-count videos, the regime where instance merging most hurts average precision.
  • Penalizing absent categories toward zero predicted count suppresses background and false detections; the paper's ablation attributes 1.5 mAP to this term.
  • Joint training also improves action classification, reaching 86.9 mAP on THUMOS14, so the same model serves both recognition and localization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the count term is mostly a regularizer, a cheap way to test this is to replace true counts with a learned count-prediction head trained on a small labeled subset; most of the gain persisting would mean per-video counts do not need to be manually annotated.
  • The attention-weighted center loss is not tied to action localization; it could be applied to other multi-label temporal tasks such as sound event detection or gesture spotting, where category presence is known but instance boundaries are costly.
  • On ActivityNet 1.2 the count term adds much less than on THUMOS14, consistent with the paper's observation of fewer multi-instance videos; this predicts that datasets with many repeated actions will show the largest benefit from count supervision.
  • Because the counting module itself has no learnable parameters, a natural ablation is to replace it with a penalty on the temporal derivative of the T-CAM; if the two behave similarly, the gain is smoothness pressure rather than information about instance number.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes 3C-Net for weakly-supervised temporal action localization using video-level class labels and action instance counts. The architecture is a two-stream I3D-based network with three losses: a classification loss on top-k pooled T-CAMs, a multi-label center loss with attention-based feature aggregation, and a counting loss. Experiments on THUMOS14 and ActivityNet 1.2 report state-of-the-art results, with ablations showing progressive improvements from each term. The main claimed novelty is the counting loss, which is intended to separate adjacent action instances by matching a predicted count to the ground-truth count.

Significance. If the counting mechanism were correctly formulated, the idea of using instance counts as weak supervision for temporal action localization would be a useful contribution, especially since count labels are cheap relative to frame-level annotations. The multi-label adaptation of the center loss is also a sensible extension. The paper provides source code, thorough ablations, and comparisons on two benchmarks. However, the central counting loss as defined is mathematically a weighted average rather than a count, so the claimed mechanistic explanation is not supported; the empirical gains may still hold but require a corrected formulation and a re-evaluation of the underlying claim.

major comments (3)
  1. [Section 3.4, Eq. (9)] Equation (9) defines m_i(j) = sum_t a^F_i(t,j) C^F_i(t,j). Since a^F_i(t,j) is a temporal softmax over t, sum_t a^F_i(t,j) = 1, so m_i(j) is a weighted average of the T-CAM values over time. A weighted average cannot represent an instance count: one can rescale all C^F_i(t,j) and obtain any desired average, so a single-instance video and a fifteen-instance video can have identical m_i(j). Consequently, the counting loss in Eq. (10) does not force the activation mass to match the count; it forces a normalized average toward the integer label. This invalidates the claim in Section 1 that 'the prediction scores sum up to a positive value within action instances and zero otherwise' and the explanation that the loss delineates adjacent instances. The observed improvement in Table 4 (26.6 vs. 24.6 mAP) may be a regularization effect, but it is not evidence for count-based delineation.
  2. [Section 3.4, density-map claim] The paper states that the attention-weighted T-CAM is 'equivalent to a density map' and that its summation yields the predicted count. This analogy with object counting [6] is unsupported: in density-based counting, the density map is a spatial map whose integral equals the count, whereas here C^F is a class score produced by a classifier and the softmax weights are normalized to sum to one. Therefore the quantity in Eq. (9) is not an integral of a density. Furthermore, the claim that 'temporal attention weighting ignores the background video segments' is not implemented, because the softmax attention assigns nonzero (albeit small) weights to all segments and no threshold is applied in the counting module. The authors should either redefine m_i as a true sum over T-CAM values (with hard attention masking) or remove the density-map/count interpretation.
  3. [Section 4.3 and Table 4] The counting loss is computed as a per-video, per-class scalar after summing over time, so it cannot by itself separate adjacent instances; it provides only a global constraint on the T-CAM. The paper does not report any direct evidence that the counting loss improves instance-level segmentation, such as the number of detected instances versus the ground-truth count, or that it is responsible for the qualitative delineation shown in Fig. 1. The ablation in Table 4 isolates the loss, but it does not test the proposed mechanism. Without such evidence, the claim that the counting loss 'delineates adjacent action instances' is not established.
minor comments (5)
  1. [Section 3.4] There is a typo: 'disrcriminative' should be 'discriminative'.
  2. [Figure 1 caption] The caption reads 'PoleVaultaction'; it should be 'PoleVault action'.
  3. [Equation (7)] The denominator '1 + sum_i y_i(j)' is unusual; standard center updates divide by the number of samples in the class. Please clarify whether this is intentional to avoid division by zero, and explain the impact on the center update.
  4. [Equation (10)] The notation n_i(j) is introduced as the ground-truth count, but earlier y_i is a binary multi-hot label. Please define n_i explicitly and state its relationship to y_i.
  5. [Section 4.1] The description of the detection threshold η for ActivityNet 1.2 is terse; please state explicitly that η is set to 0 for that dataset and how the detection algorithm then generates segments from the T-CAM.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the count and center losses are supervised by external video-level labels and evaluated on held-out benchmarks, so no claimed result reduces to its own input.

full rationale

The derivation is self-contained and non-circular. The classification loss uses video-level multi-hot category labels; the center loss learns class centers from features aggregated by attention and is penalized against those external labels; the counting loss compares the model's predicted count m_i(j) (Eq. 9) with ground-truth counts n_i(j) (Eq. 10) that are generated from temporal segment annotations on the training split. Neither the predicted count nor the localization output is fitted to the evaluation target: localization is measured by mAP on held-out THUMOS14 test and ActivityNet 1.2 validation videos. The only self-citation is reference [6], used to support the density-map analogy for attention-weighted activations; that prior object-counting work is independently published and evaluated, so it does not smuggle in the present paper's conclusion. The reader's Eq. 9 concern is a legitimate mathematical correctness issue: because the temporal softmax in Eq. 4 sums to one over time, m_i(j) is a weighted average of T-CAM values rather than an instance count. That, however, is not a circularity: the prediction is still computed from the model and supervised by external count labels, and the claimed localization gains are empirically evaluated rather than derived from the count-loss definition.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four assumptions: fixed I3D features are sufficient, Eq. 9 produces a count, counts are annotatable without temporal boundaries, and T-CAM thresholding yields detections. The Eq. 9 count premise is the most fragile because it is mathematically a weighted average, not a count. The count-label assumption is also weakened by the benchmarking procedure, which derives counts from temporal segment annotations.

free parameters (4)
  • alpha (center loss weight) = 1e-3
    Set by hand for both datasets; controls the influence of the center loss in Eq. 1.
  • beta (counting loss weight) = 1 (THUMOS14), 0.1 (ActivityNet 1.2)
    Tuned per dataset; directly controls the counting term that the paper credits for the largest performance jump.
  • lambda (negative-count loss weight) = 1e-3
    Set by hand to balance positive and negative action categories in Eq. 10.
  • top-k pooling ratio = k = ceil(si/8)
    Heuristic for classification pooling in Section 3.2; affects the category scores used for detection.
assumptions (4)
  • domain assumption Fixed pretrained I3D features from Kinetics are a sufficient representation for action localization.
    Section 3, Feature Extraction: both streams use frozen Kinetics-pretrained I3D features. If these features lack localization information, no learned loss can recover it.
  • ad hoc to paper The attention-weighted sum in Eq. 9 yields an action-instance count.
    Section 3.4: because a^F is a softmax over time, sum_t a^F(t,j)=1, so the quantity is a weighted average, not a count. The counting loss relies on this premise.
  • domain assumption Ground-truth action counts can be annotated independently of temporal boundaries.
    Section 4.1, Count Labels: counts were generated from temporal segment annotations, while the paper asserts independent annotation is possible. The experiments do not test that assertion.
  • domain assumption Thresholding the T-CAM produces valid action instance detections.
    Section 3.5: inference extracts detections where the T-CAM crosses threshold eta. This is the standard assumption in T-CAM-based weakly supervised localization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization." pith.science (2026). https://pith.science/paper/6ULEADP3

@misc{pith2026190808216,
  author       = {Pith},
  title        = {Pith review of: 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ULEADP3}},
  note         = {Machine review of arXiv:1908.08216}
}
read the original abstract

Temporal action localization is a challenging computer vision problem with numerous real-world applications. Most existing methods require laborious frame-level supervision to train action localization models. In this work, we propose a framework, called 3C-Net, which only requires video-level supervision (weak supervision) in the form of action category labels and the corresponding count. We introduce a novel formulation to learn discriminative action features with enhanced localization capabilities. Our joint formulation has three terms: a classification term to ensure the separability of learned action features, an adapted multi-label center loss term to enhance the action feature discriminability and a counting loss term to delineate adjacent action sequences, leading to improved localization. Comprehensive experiments are performed on two challenging benchmarks: THUMOS14 and ActivityNet 1.2. Our approach sets a new state-of-the-art for weakly-supervised temporal action localization on both datasets. On the THUMOS14 dataset, the proposed method achieves an absolute gain of 4.6% in terms of mean average precision (mAP), compared to the state-of-the-art. Source code is available at https://github.com/naraysa/3c-net.

Figures

Figures reproduced from arXiv: 1908.08216 by the authors.

Figure 1
Figure 1. Predicted action proposals for a video clip containing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our overall architecture (3C-Net) with different loss terms (classification, center and counting), and the associated modules. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation study with respect to difference design choices [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative temporal action localization results of our 3C-Net approach on example videos from the THUMOS14 and ActivityNet [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 31 canonical work pages

  1. [6]

    Object counting and instance segmentation with image-level supervision

    Hisham Cholakkal, Guolei Sun, Fahad Shahbaz Khan, and Ling Shao. Object counting and instance segmentation with image-level supervision. In CVPR, 2019. 2, 5

  2. [1]

    Finding actors and actions in movies

    Piotr Bojanowski, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Finding actors and actions in movies. In ICCV, 2013. 1, 2

  3. [2]

    Weakly su- pervised action labeling in videos under ordering constraints

    Piotr Bojanowski, R ´emi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly su- pervised action labeling in videos under ordering constraints. In ECCV, 2014. 2

  4. [3]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR,

  5. [4]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR,

  6. [5]

    Re- thinking the faster r-cnn architecture for temporal action lo- calization

    Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Sey- bold, David A Ross, Jia Deng, and Rahul Sukthankar. Re- thinking the faster r-cnn architecture for temporal action lo- calization. In CVPR, 2018. 1, 6

  7. [7]

    Automatic annotation of human actions in video

    Olivier Duchenne, Ivan Laptev, Josef Sivic, Francis Bach, and Jean Ponce. Automatic annotation of human actions in video. In ICCV, 2009. 2

  8. [8]

    C-wsl: Count-guided weakly supervised lo- calization

    Mingfei Gao, Ang Li, Ruichi Yu, Vlad I Morariu, and Larry S Davis. C-wsl: Count-guided weakly supervised lo- calization. In ECCV, 2018. 2

Show all 34 references
  1. [9]

    The thu- mos challenge on action recognition for videos in the wild

    Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thu- mos challenge on action recognition for videos in the wild. CVIU, 2017. 2, 6

  2. [10]

    What do 15,000 object categories tell us about classifying and lo- calizing actions? In CVPR, 2015

    Mihir Jain, Jan C Van Gemert, and Cees GM Snoek. What do 15,000 object categories tell us about classifying and lo- calizing actions? In CVPR, 2015. 7

  3. [11]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

  4. [12]

    Learning realistic human actions from movies

    Ivan Laptev, Marcin Marszałek, Cordelia Schmid, and Ben- jamin Rozenfeld. Learning realistic human actions from movies. In CVPR, 2008. 1, 2

  5. [13]

    Spot on: Action localization from pointly-supervised proposals

    Pascal Mettes, Jan C Van Gemert, and Cees GM Snoek. Spot on: Action localization from pointly-supervised proposals. In ECCV, 2016. 1, 2

  6. [14]

    Weakly supervised action localization by sparse temporal pooling network

    Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. Weakly supervised action localization by sparse temporal pooling network. In CVPR, 2018. 1, 2, 3, 6

  7. [15]

    The lear submission at thumos 2014

    Dan Oneata, Jakob Verbeek, and Cordelia Schmid. The lear submission at thumos 2014. 2013. 1, 6

  8. [16]

    W- talc: Weakly-supervised temporal activity localization and classification

    Sujoy Paul, Sourya Roy, and Amit K Roy-Chowdhury. W- talc: Weakly-supervised temporal activity localization and classification. In ECCV, 2018. 1, 3, 5, 6, 7

  9. [17]

    Weakly supervised action learning with rnn based fine-to-coarse modeling

    Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In CVPR, 2017. 2

  10. [18]

    Modeling the temporal ex- tent of actions

    Scott Satkin and Martial Hebert. Modeling the temporal ex- tent of actions. In ECCV, 2010. 1

  11. [19]

    Smoothing and differentiation of data by simplified least squares procedures

    Abraham Savitzky and Marcel JE Golay. Smoothing and differentiation of data by simplified least squares procedures. Analytical chemistry, 36(8):1627–1639, 1964. 6

  12. [20]

    Action snippets: How many frames does human action recognition require? In CVPR, 2008

    Konrad Schindler and Luc Van Gool. Action snippets: How many frames does human action recognition require? In CVPR, 2008. 1

  13. [21]

    Cdc: Convolutional-de- convolutional networks for precise temporal action localiza- tion in untrimmed videos

    Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. Cdc: Convolutional-de- convolutional networks for precise temporal action localiza- tion in untrimmed videos. In CVPR, 2017. 1, 6

  14. [22]

    Autoloc: weakly-supervised temporal action localization in untrimmed videos

    Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang. Autoloc: weakly-supervised temporal action localization in untrimmed videos. In ECCV, 2018. 6, 7

  15. [23]

    Temporal action localization in untrimmed videos via multi-stage cnns

    Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In CVPR, 2016. 1, 6

  16. [24]

    Two-stream con- volutional networks for action recognition in videos

    Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In NIPS, 2014. 7

  17. [25]

    Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization

    Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In ICCV, 2017. 1, 2

  18. [26]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015. 7

  19. [27]

    Action recognition with improved trajectories

    Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV, 2013. 7

  20. [28]

    Untrimmednets for weakly supervised action recognition and detection

    Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In CVPR, 2017. 1, 2, 3, 5, 6, 7

  21. [29]

    Temporal segment networks: Towards good practices for deep action recogni- tion

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, 2016. 3, 6, 7

  22. [30]

    A discriminative feature learning approach for deep face recog- nition

    Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recog- nition. In ECCV, 2016. 1, 3, 4, 5

  23. [31]

    R-c3d: Region convolutional 3d network for temporal activity detection

    Huijuan Xu, Abir Das, and Kate Saenko. R-c3d: Region convolutional 3d network for temporal activity detection. In ICCV, 2017. 1, 6

  24. [32]

    A du- ality based approach for realtime tv-l 1 optical flow

    Christopher Zach, Thomas Pock, and Horst Bischof. A du- ality based approach for realtime tv-l 1 optical flow. In Joint pattern recognition symposium, 2007. 6

  25. [33]

    Temporal action detection with structured segment networks

    Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xi- aoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, 2017. 1, 6, 7

  26. [34]

    Learning deep features for discrimi- native localization

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discrimi- native localization. In CVPR, 2016. 3 9

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.