REVIEW 3 major objections 5 minor 58 references
Scale Matters: Temporal Scale Aggregation Network for Precise Action Localization in Untrimmed Videos
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read TSA-Net claims that ensembling temporal convolutions with different dilation rates—organized into branches tuned to short, middle, and long actions—plus start/mid/end point pairing, sets new state-of-the-art action localization results on…
desk verdict The architecture is solid and the controlled comparisons support it, but the headline THUMOS14 record compares P3D+UNet against an I3D baseline and overstates what TSA-Net alone delivers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The multi-dilation temporal convolution (MDC) block is the core object: within one block, three 1-D temporal convolutions share a kernel size but use dilation rates $d_1 < d_2 < d_3$, their outputs are averaged, and a skip connection is added so the block learns a residual. Two MDC blocks are stacked in each of three branches with dilation schedules (1,2,3), (1,3,5), and (1,5,7), giving receptive fields of 13 to 29 snippets, so each branch is receptive to a different action-duration scale. Branch outputs are average-pooled before a small predictor emits start, midpoint, and end probability sequences; proposals are formed by pairing start and end points, gated by the midpoint confidence, and ranked by a learned Bayesian compatibility function that scores a proposal from 96 sampled values of the three probability sequences.
What would settle it
Train and evaluate TSA-Net on THUMOS14 with I3D features and compare mAP@0.5 with the prior best 42.8%; if the gap nearly disappears, the headline record is feature-driven rather than caused by the multi-scale architecture.
Extended reading notes
Core claim
The paper's central claim is that boundary precision in two-stage temporal action localization is limited by a fixed temporal receptive field, and that ensembling dilation rates fixes it. TSA-Net detects three kinds of critical points—starting, midpoint, and ending—and pairs start/end candidates only when the detected midpoint is also confident, which the authors argue removes low-accuracy proposals that two-point boundary methods produce. The proposal generator uses parallel branches of stacked MDC blocks with different dilation schedules, each branch specialized to a duration range, and merges their outputs by average pooling before predicting critical-point probability sequences. On identical two-stream features, TSA-Net improves THUMOS14 AR@50 from 37.46% to 42.83% and mAP@0.5 from 36.9% to 41.5%; with P3D features it reports 46.9% mAP@0.5, and on ActivityNet-1.3 it raises average mAP from 32.84% to 34.62%.
Load-bearing premise
The headline comparison assumes that a result obtained with P3D features can be fairly compared with a previous result obtained with I3D features, so the claimed margin could reflect feature differences rather than the network alone.
Editorial extensions
If this is right
- Higher precision at the same proposal budget: on THUMOS14 with two-stream features, AR@50 rises from 37.46% to 42.83%, so a video system can retrieve more true action segments with fewer candidate proposals.
- Both short and long actions gain: the duration-stratified evaluation reports improvement on every quintile of action durations, not just the middle range.
- Unseen classes transfer: on ActivityNet-1.3, AUC and AR@100 drop only slightly for action classes never seen in training, suggesting the scale-aggregation mechanism is not class-specific.
- The MDC block is the source of the gain: replacing MDC blocks with parameter-matched standard 1-D convolutions drops AR@50 from 40.83 to 31.48, and any single dilation branch underperforms the three-branch ensemble.
Reading between the lines
- The three-point design suggests a direct extension to streaming localization: a confident midpoint could act as a trigger, with a proposal committed only when a later ending point lands, reducing false positives in live video monitoring.
- The paper only tests fixed dilation schedules; a natural next probe is to learn branch dilation rates or branch weights per video, to see whether the fixed schedules are near-optimal or merely adequate.
- The same scale-aggregation reasoning may transfer to temporal action segmentation and dense video captioning, where boundary precision is also the limiting factor and where MDC blocks could replace larger kernels with cheaper dilated ones.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes TSA-Net, a two-stage temporal action localization method. Its proposal-generation stage detects starting, mid, and ending critical points using multi-dilation temporal convolution (MDC) blocks arranged in three branches with different dilation rates, followed by average pooling; proposals are formed by pairing starting/ending points with mid-point filtering and scored by a learned compatibility function in a Bayesian-style product. The second stage reuses existing classifiers (UNet, SCNN-cls, and the classifier of [55]). The authors evaluate on THUMOS14 and ActivityNet-1.3 and claim state-of-the-art results, e.g., 46.9% mAP@0.5 on THUMOS14 versus 42.8% for the previous best.
Significance. The MDC block and multi-branch scale aggregation are simple and well-motivated, and the controlled experiments (two-stream features, same classifier) show consistent gains over BSN on both proposal generation (e.g., AR@50 from 37.46 to 42.83 with Soft-NMS) and final localization (mAP@0.5 from 36.9 to 41.5 with UNet). The ActivityNet comparison is also controlled and positive. If the reported gains are reproducible, the paper makes a useful contribution to temporal action localization. However, the headline claim of a new state-of-the-art record on THUMOS14 is not supported by a matched comparison, and some internal inconsistencies and missing statistical details reduce confidence.
major comments (3)
- [Abstract, §4.5, Table 7] The claim that TSA-Net 're-calibrates new state-of-the-art' on THUMOS14 is based on a comparison that changes two variables at once. The 46.9% mAP@0.5 is obtained with P3D features and the UNet classifier, whereas the quoted previous best of 42.8% (TAL-Net) uses I3D features and its own classifier. The same table shows a controlled comparison (Ours Two-Stream + UNet 41.5 vs BSN + UNet 36.9) that supports the architecture, but the headline margin of 4.1 points overstates the contribution. Please either compare with the same features/classifier or rephrase the state-of-the-art claim to identify the matched baselines.
- [§4.3, §4.4, Tables 1, 4, 5, 6] The AR values for the same configuration are inconsistent across tables. Table 1 reports Ours+Greedy-NMS at AR@50 as 41.40 and Ours+Soft-NMS as 42.83, while Tables 4-6 repeatedly report TSA-Net/multi-branch at AR@50 as 40.83. The text says 'Bayesian scores are used to rank proposals' in §4.4, but Table 6 indicates that the 40.83 value corresponds to the product P(s)P(e) without the compatibility function, while the full Bayesian score including φ gives 42.83. Please clarify precisely which scoring and NMS variant is used in each table, as the current presentation makes the ablation numbers difficult to reconcile.
- [§4.2, Tables 2 and 8] No variance estimates or multiple-seed results are provided. The ActivityNet gains are small (AUC 67.01 vs 66.26 on the test set; average mAP 34.62 vs 32.84), and without repeated runs or error bars it is not possible to assess whether the 'clear and consistent' improvement is statistically meaningful. At minimum, report mean and standard deviation over several training runs for the main comparisons.
minor comments (5)
- [§3.2] The phrase 'sequence-to-sequence learning' and the citation to [20] are not illuminating; the model is a per-frame classification network. Consider removing or rewriting the sentence.
- [§3.3, Eq. (2)] Eq. (2) is called a Bayesian formulation, but it is a product of independently estimated probabilities multiplied by a learned compatibility score; please state that it is a heuristic scoring function.
- [Table 7] Table 7 lists IoU thresholds in descending order (0.7, 0.6, ..., 0.3); consider ordering them ascending for consistency with the text (0.3 to 0.7).
- [§4.1] The frame sampling strides (5 on THUMOS14, 16 on ActivityNet) are introduced as choices balancing complexity and fidelity, but no sensitivity analysis is given.
- [Throughout] Minor typos: 'staring' in the introduction; the phrase 'starting / mid-point / ending' appears with inconsistent hyphens in the abstract.
Circularity Check
No circularity: the paper's claims are empirical evaluations on held-out benchmarks; the headline comparison is confounded but not circular.
full rationale
The paper makes no derivational claim that reduces to its own inputs. The central method is an architecture (MDC blocks, multi-branch stacking, three-point critical detection) trained with cross-entropy and ranking losses on standard training splits of THUMOS14 and ActivityNet-1.3, then evaluated on held-out test data (THUMOS14 test set and ActivityNet-1.3 server). The Bayesian proposal ranking in Eq. (2) is a product of learned point-wise probabilities and a separately trained compatibility function phi; phi is learned in Eq. (3) by regression to IoU with ground-truth segments. This is ordinary supervised training, not a recycled prediction: no fitted parameter is later reported as a prediction on the same data. The ablation studies compare variants on a held-out test set, and the final mAP/AR numbers come from applying the trained model to unseen videos. No load-bearing self-citation is present: the paper draws on prior work such as BSN, SSN, and TAL-Net for standard components, but the claimed novelty (multi-dilation temporal convolution blocks, multi-branch scale aggregation, and mid-point pairing) is implemented and evaluated here rather than imported from a same-author citation. The one notable weakness is a comparison confound: the headline THUMOS14 46.9% uses P3D features and the UntrimmedNet classifier, while the quoted previous best of 42.8% (TAL-Net) uses I3D features and its own classifier, so the 4.1-point gap is not purely attributable to TSA-Net. That is a correctness/fairness issue, not circularity. The same-feature comparisons (Ours 41.5 vs BSN 36.9 at mAP@0.5 on two-stream + UNet; ActivityNet test average 34.62 vs BSN 32.84 with identical features) provide independent, more controlled support for the architecture. Verdict: no significant circularity.
Assumptions & free parameters
free parameters (6)
- Dilation rate sets for MDC branches =
(1,2,3), (1,3,5), (1,5,7)
- Critical-point label inflation delta =
0.1
- Critical point selection threshold =
0.9
- Proposal duration bounds [dmin, dmax] =
estimated from training set annotations
- Segment extension factor =
1.4
- Frame sampling stride =
5 on THUMOS14, 16 on ActivityNet-1.3
assumptions (4)
- domain assumption Temporal context of the correct scale is necessary to estimate boundary and midpoint points.
- ad hoc to paper Average pooling over three branches with fixed dilation sets is an effective fusion for scale aggregation.
- domain assumption Pre-extracted features (two-stream, P3D, C3D) contain enough signal for boundary estimation.
- domain assumption Published results with differing feature backbones are comparable benchmark numbers.
Cite this review
Pith. "Pith review of Scale Matters: Temporal Scale Aggregation Network for Precise Action Localization in Untrimmed Videos." pith.science (2026). https://pith.science/paper/5WRJUVGN
@misc{pith2026190800707,
author = {Pith},
title = {Pith review of: Scale Matters: Temporal Scale Aggregation Network for Precise Action Localization in Untrimmed Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/5WRJUVGN}},
note = {Machine review of arXiv:1908.00707}
}
read the original abstract
Temporal action localization is a recently-emerging task, aiming to localize video segments from untrimmed videos that contain specific actions. Despite the remarkable recent progress, most two-stage action localization methods still suffer from imprecise temporal boundaries of action proposals. This work proposes a novel integrated temporal scale aggregation network (TSA-Net). Our main insight is that ensembling convolution filters with different dilation rates can effectively enlarge the receptive field with low computational cost, which inspires us to devise multi-dilation temporal convolution (MDC) block. Furthermore, to tackle video action instances with different durations, TSA-Net consists of multiple branches of sub-networks. Each of them adopts stacked MDC blocks with different dilation parameters, accomplishing a temporal receptive field specially optimized for specific-duration actions. We follow the formulation of boundary point detection, novelly detecting three kinds of critical points (ie, starting / mid-point / ending) and pairing them for proposal generation. Comprehensive evaluations are conducted on two challenging video benchmarks, THUMOS14 and ActivityNet-1.3. Our proposed TSA-Net demonstrates clear and consistent better performances and re-calibrates new state-of-the-art on both benchmarks. For example, our new record on THUMOS14 is 46.9% while the previous best is 42.8% under mAP@0.5.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[55]
Y . Zhao, B. Zhang, Z. Wu, S. Yang, L. Zhou, S. Yan, L. Wang, Y . Xiong, D. Lin, Y . Qiao, et al. Cuhk & ethz & siat submission to activitynet challenge 2017. arXiv preprint arXiv:1710.08011, 2017. 5, 8
arXiv 2017
-
[1]
H. Alwassel, F. C. Heilbron, and B. Ghanem. Action search: Spotting actions in videos and its application to temporal action localization. In ECCV, 2018. 1
work page 2018
- [2]
-
[3]
S. Buch, V . Escorcia, B. Ghanem, L. Fei-Fei, and J. C. Niebles. End-to-end, single-stream temporal action detection in untrimmed videos. In BMVC, 2017. 2
work page 2017
-
[4]
S. Buch, V . Escorcia, C. Shen, B. Ghanem, and J. C. Niebles. SST: single-stream temporal action proposals. In CVPR, 2017. 2, 6, 8
work page 2017
-
[5]
J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, 2017. 6, 8
work page 2017
-
[6]
Y . Chao, S. Vijayanarasimhan, B. Seybold, D. A. Ross, J. Deng, and R. Sukthankar. Rethinking the faster R-CNN architecture for temporal action localization. In CVPR, 2018. 2, 3, 8
work page 2018
-
[7]
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834–848, 2018. 3
work page 2018
Show all 58 references
-
[8]
X. Dai, B. Singh, G. Zhang, L. S. Davis, and Y . Q. Chen. Temporal context network for activity localization in videos. In ICCV, 2017. 2, 7, 8
2017
-
[9]
Escorcia, F
V . Escorcia, F. C. Heilbron, J. C. Niebles, and B. Ghanem. Daps: Deep action proposals for action understanding. In ECCV, 2016. 2, 6
2016
-
[10]
J. Gao, K. Chen, and R. Nevatia. CTAP: complementary temporal action proposal generation. In ECCV, 2018. 2, 6, 7, 8
2018
-
[11]
J. Gao, Z. Yang, and R. Nevatia. Cascaded boundary regression for temporal action detection. In BMVC,
-
[12]
J. Gao, Z. Yang, C. Sun, K. Chen, and R. Nevatia. TURN TAP: temporal unit regression network for temporal action proposals. In ICCV, 2017. 2, 6, 8
2017
-
[13]
R. B. Girshick. Fast R-CNN. In ICCV, 2015. 2, 5
2015
-
[14]
R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 1, 2
2014
-
[15]
F. C. Heilbron, W. Barrios, V . Escorcia, and B. Ghanem. SCC: semantic context cascade for efficient action detection. In CVPR, 2017. 8
2017
-
[16]
F. C. Heilbron, V . Escorcia, B. Ghanem, and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015. 6
2015
-
[17]
F. C. Heilbron, J. Lee, H. Jin, and B. Ghanem. What do I annotate next? an empirical study of active learning for action localization. In ECCV, 2018. 1
2018
-
[18]
F. C. Heilbron, J. C. Niebles, and B. Ghanem. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos. In CVPR, pages 1914–1923, 2016. 2
1914
-
[19]
Idrees, A
H. Idrees, A. R. Zamir, Y . Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah. The thumos challenge on action recognition for videos in the wild.Computer Vision and Image Understanding, 155:1–23, 2017. 5
2017
-
[20]
Isola, J
P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image- to-image translation with conditional adversarial networks. In CVPR, 2017. 4
2017
-
[21]
Krishna, K
R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles. Dense-captioning events in videos. In ICCV,
-
[22]
W. Kuo, B. Hariharan, and J. Malik. Deepbox: Learning objectness with convolutional networks. In ICCV, 2015. 2
2015
-
[23]
C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In CVPR, 2017. 3
2017
-
[24]
Y . Li, X. Zhang, and D. Chen. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In CVPR, 2018. 3
2018
-
[25]
T. Lin, P. Doll ´ar, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 2
2017
-
[26]
T. Lin, X. Zhao, and Z. Shou. Single shot temporal action detection. In ACM Multimedia, 2017. 2
2017
-
[27]
T. Lin, X. Zhao, and Z. Shou. Temporal convolution based action proposal: Submission to activitynet
-
[28]
T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang. BSN: boundary sensitive network for temporal action proposal generation. In ECCV, 2018. 2, 3, 5, 6, 7, 8
2018
-
[29]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, and A. C. Berg. SSD: single shot multibox detector. In ECCV, 2016. 2
2016
-
[30]
Mehta, M
S. Mehta, M. Rastegari, A. Caspi, L. G. Shapiro, and H. Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In ECCV, 2018. 3
2018
-
[31]
Z. Qiu, T. Yao, and T. Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, 2017. 3, 6
2017
-
[32]
Redmon, S
J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016. 2 9
2016
-
[33]
S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R- CNN: towards real-time object detection with region proposal networks. In NeurIPS, 2015. 2
2015
-
[34]
Z. Shou, J. Chan, A. Zareian, K. Miyazawa, and S. Chang. CDC: convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. In CVPR, 2017. 1, 3, 8
2017
-
[35]
Z. Shou, H. Gao, L. Zhang, K. Miyazawa, and S. Chang. Autoloc: Weakly-supervised temporal action localization in untrimmed videos. In ECCV,
-
[36]
Z. Shou, D. Wang, and S. Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In CVPR, 2016. 1, 2, 5, 6, 8
2016
-
[37]
Simonyan and A
K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014. 3
2014
-
[38]
H. Song, W. Wang, S. Zhao, J. Shen, and K. Lam. Pyramid dilated deeper convlstm for video salient object detection. In ECCV, 2018. 3
2018
-
[39]
Soomro, A
K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. 6
2012 arXiv
-
[40]
H. Su, X. Zhao, and T. Lin. Cascaded pyramid mining network for weakly supervised temporal action localization. CoRR, abs/1810.11794, 2018. 1
2018 arXiv
-
[41]
D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015. 3, 6
2015
-
[42]
J. Wang, W. Jiang, L. Ma, W. Liu, and Y . Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR, 2018. 1
2018
-
[43]
L. Wang, Y . Xiong, D. Lin, and L. V . Gool. Untrimmednets for weakly supervised action recogni- tion and detection. In CVPR, 2017. 5, 8
2017
-
[44]
Y . Wei, H. Xiao, H. Shi, Z. Jie, J. Feng, and T. S. Huang. Revisiting dilated convolution: A simple approach for weakly- and semi-supervised semantic segmentation. In CVPR, 2018. 3
2018
-
[45]
Xiong, L
Y . Xiong, L. Wang, Z. Wang, B. Zhang, H. Song, W. Li, D. Lin, Y . Qiao, L. V . Gool, and X. Tang. CUHK & ETHZ & SIAT submission to activitynet challenge 2016. CoRR, abs/1608.00797, 2016. 6
2016 arXiv
-
[46]
Xiong, Y
Y . Xiong, Y . Zhao, L. Wang, D. Lin, and X. Tang. A pursuit of temporal accuracy in general activity detection. CoRR, abs/1703.02716, 2017. 6, 7
2017 arXiv
-
[47]
B. Xu, H. Ye, Y . Zheng, H. Wang, T. Luwang, and Y . Jiang. Dense dilated network for few shot action recognition. In ICMR, 2018. 3
2018
-
[48]
H. Xu, A. Das, and K. Saenko. R-C3D: region convolutional 3d network for temporal activity detection. In ICCV, 2017. 1, 2, 8
2017
-
[49]
Y . Xu, C. Zhang, Z. Cheng, J. Xie, Y . Niu, S. Pu, and F. Wu. Segregated temporal assembly recurrent networks for weakly supervised multiple action detection. CoRR, abs/1811.07460, 2018. 1
2018 arXiv
-
[50]
Yeung, O
S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei. End-to-end learning of action detection from frame glimpses in videos. In CVPR, 2016. 2
2016
-
[51]
Yu and V
F. Yu and V . Koltun. Multi-scale context aggregation by dilated convolutions. CoRR, abs/1511.07122,
-
[52]
F. Yu, V . Koltun, and T. A. Funkhouser. Dilated residual networks. In CVPR, 2017. 3
2017
-
[53]
Zhang, X
D. Zhang, X. Dai, X. Wang, and Y . Wang. S3D: single shot multi-span detector via fully 3d convolutional networks. In BMVC, 2018. 2
2018
-
[54]
Y . Zhao, Y . Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin. Temporal action detection with structured segment networks. In ICCV, 2017. 1, 2, 3, 5, 8
2017
-
[56]
Zhong, N
J. Zhong, N. Li, W. Kong, T. Zhang, T. H. Li, and G. Li. Step-by-step erasion, one-by-one collection: A weakly supervised temporal action detector. In ACM Multimedia, 2018. 1
2018
-
[57]
L. Zhou, Y . Zhou, J. J. Corso, R. Socher, and C. Xiong. End-to-end dense video captioning with masked transformer. In CVPR, 2018. 1 10
2018
-
[2017]
CoRR, abs/1707.06750, 2017. 7
2017 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.