Relation-Aware Pyramid Network (RapNet) for temporal action proposal

Jialin Gao; Jiani Li; Jiwei Li; Xi Zhou; Yufeng Yuan; Zhixiang Shi

REVIEW 3 major objections 5 minor 11 references

Relation-Aware Pyramid Network (RapNet) for temporal action proposal

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a temporal action proposal pipeline built around RapNet—a relation-aware pyramid network with self-attention and FPN—plus a two-stage boundary adjustment scheme and ensemble, reaches 71.51% validation and 71.38%…

desk verdict A competent, internally consistent challenge report whose named novelty (self-attention + FPN) is never ablated, so the reported gains cannot be attributed to the proposed design. read the letter →

arxiv 1908.03448 v1 pith:FK4FG2PL submitted 2019-08-09 cs.CV

Jialin Gao , Zhixiang Shi , Jiani Li , Yufeng Yuan , Jiwei Li , Xi Zhou This is my paper · ORCID

classification cs.CV

keywords temporalactionproposalRapNetself-attentionfeaturepyramidnetworkboundaryadjustmentActivityChallengevideounderstandingensemble

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a competition report for temporal action proposal, the task of generating candidate time intervals that likely contain actions. Its central claim is that a specific pipeline—RapNet's multiscale anchor-based proposals, followed by a two-stage boundary adjustment, then ensembling over two video encoders and two anchor sets—is a top-performing solution: the authors report 71.51% validation AUC and 71.38% test AUC, which they say placed second in ActivityNet Challenge 2019 Task 1. A sympathetic reader should care because temporal action proposals are the front end of video understanding systems, and the reported numbers are on a widely used benchmark with a public leaderboard.

What carries the argument

The central mechanism is RapNet, a temporal pyramid proposal generator: a 1D feature-pyramid network with six generators at different temporal scales, where self-attention is inserted on the top-down path to capture long-range snippet relationships and FPN on the bottom-up path supplies multiscale features. Anchors are chosen by K-means clustering over ground-truth durations, and each anchor predicts confidence, center and width, and IoU against ground truth. Proposal boundaries are then adjusted by a two-stage scheme using BSN's PEM module with frame-level actionness and TAG's watershed algorithm; redundant proposals are suppressed with soft-NMS.

What would settle it

Reproduce the recipe on ActivityNet v1.3 with the same train/validation split and compute AR@AN and AUC; if the single RapNet model does not approach the reported 69.61% validation AUC, or if the two-stage boundary adjustment does not add roughly one point of AUC, the central claim is not supported. A targeted test would bypass fixed-length resizing and feed variable-length representations to see whether the 128-snippet compression is hurting boundary recall.

Watch

Extended reading notes

Core claim

The paper reports that combining a temporal pyramid network with self-attention and FPN generates strong multiscale proposals: on ActivityNet v1.3 validation, RapNet alone gives 69.61% AUC, adding the PEM boundary-refinement stage brings it to 70.35%, and the further TAG watershed adjustment reaches 70.65% AUC. Ensembling ResNet-50 and ResNet-101 C3D backbones with 12 and 18 anchors raises the result to 71.51% validation AUC and 71.38% test-server AUC, which the authors state is second place in ActivityNet Challenge 2019 Task 1. The paper treats each boundary-adjustment stage as a measurable improvement and presents the ensemble as the final contribution.

Load-bearing premise

The pipeline assumes that resizing every video to a fixed 128-snippet representation preserves the temporal boundary cues needed to localize actions whose lengths vary widely.

Editorial extensions

If this is right

On the authors' reported numbers, the single-model pipeline already reaches 69.61% AUC, so the architecture is competitive before any ensembling.
Boundary adjustment is worth about one AUC point: PEM adds 0.74 points and the TAG watershed stage adds another 0.30 points.
Ensembling the two C3D backbones and two anchor sets adds about 0.86 AUC points over the single refined system, reaching 71.51% on validation.
The proposal generator and boundary-refinement stages are separable, meaning each component could be grafted onto other action detection systems independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not ablate self-attention, FPN, or anchor count separately, so a reader cannot attribute the AUC gains to any single RapNet component; a clean ablation is the natural next experiment.
The fixed 128-snippet representation is the most fragile part of the recipe; a variable-length or multi-resolution representation might improve boundary recall, especially for very short actions.
If the roughly one-point AUC gain from boundary adjustment is generic rather than specific to RapNet, it would transfer to other anchor-based proposal generators; that is testable by swapping base generators.
Because the test number is a single leaderboard evaluation from a competition, repeated training runs and multiple seeds would be needed to know the method's variance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a short technical report from the ActivityNet 2019 challenge, not a research paper, and it should be read that way. The system itself is a reasonable composition of known pieces: fine-tuned ResNet-C3D features, YOLO-style anchors, a temporal FPN with self-attention, BSN's PEM, TAG, and soft-NMS. What is new is the specific combination and the empirical result: 71.51% validation AUC and 71.38% on the challenge test server, with a claimed 2nd place. The numbers in Table 2 are internally consistent: each added post-processing component improves AR and AUC, which suggests the pipeline was not tuned to the test set. The paper also gives enough detail on anchors, loss weights, and label assignment that a reader could reproduce the approach in broad terms.

The main weakness is exactly where the stress-test note lands. The abstract and Section 2 sell RapNet as a relation-aware proposal network, with self-attention on the top-down path and a 1D FPN on the bottom-up path. But Table 2 only ablates APG, +PEM, and +TAG. It never removes the self-attention or the FPN, so 69.61% could come from the anchor design, the features, or the FPN while the self-attention adds little or nothing. That is a real gap. It is not a reason to call the benchmark numbers false; it is a reason to say the central attribution claim is unsupported by the evidence presented. If I had to guess, the relation-aware modules probably do help a bit, but the paper doesn't let me defend that bet.

Other soft spots are smaller. There are no error bars, no training details like batch size, learning rate, epoch count, or snippet length L, and no comparison against published state-of-the-art methods (BMN, DBG, etc.). The fixed 128-snippet resizing is a design choice that could hurt performance on very long or very short actions, but the paper never claims to be optimal on that axis, so it is a minor concern. The test result comes from an external server, which is good; the lack of released code means the result is not independently re-runnable, but for a challenge report that is par for the course.

Overall, this is a solid, useful technical report for someone building a temporal action proposal pipeline out of existing components. It is not a conceptual breakthrough and it is not a proof that relation-aware modules work. A serious editor would not desk-reject it outright if the venue is a workshop or a challenge-focused track; for a top conference or journal, the missing ablation alone would force major revision or rejection. I would bring it to a reading group as an example of an effective challenge submission, and I would encourage the authors to release code and run the missing ablation, but I would not cite it as evidence for the efficacy of self-attention in temporal proposals without that follow-up.

Referee Report

3 major / 5 minor

Summary. This manuscript is a technical report describing the authors' submission to the ActivityNet Challenge 2019 temporal action proposal task. The proposed system, Relation-Aware Pyramid Network (RapNet), extracts snippet-level features with a fine-tuned ResNet-50/101 C3D backbone, resizes all videos to a fixed 128 x 256 feature map, and generates multiscale anchor-based proposals using a temporal feature pyramid network enhanced with self-attention modules. A two-stage boundary adjustment scheme (refined PEM followed by soft-NMS and watershed-based TAG refinement) is applied, and an ensemble over two backbones and two anchor configurations is used. The authors report 70.65% validation AUC for the single model with boundary adjustment, 71.51% validation AUC after ensemble, and 71.38% test AUC, claiming this result helped them achieve 2nd place in the challenge.

Significance. If the reported numbers hold, the system is clearly competitive at the top of a well-known benchmark, and the external test-server evaluation is a strength that rules out adaptation to the test set. The paper also provides an incremental ablation of the boundary-adjustment components (Table 2), and the feature-extraction and proposal-generation pipeline is described in enough detail to be broadly replicable. However, the central novelty claimed in the title and Section 2, namely the relation-aware (self-attention) and pyramid (FPN) design, is never isolated in the experiments, so the paper does not currently establish that its proposed architecture, rather than the anchor design, features, or post-processing, is responsible for the reported performance. The fixed-length resizing to 128 snippets is also a major design choice without supporting analysis. The paper is best assessed as an incomplete technical report rather than a fully supported research contribution.

major comments (3)

[Sec. 2 and Table 2] The only ablation in the paper, Table 2, compares APG, +PEM, and +TAG, but it never removes the self-attention modules or the 1D FPN from the APG baseline. Since Sections 2 and Figure 1 present self-attention and FPN as the core of RapNet, the reported 69.61% AUC for APG cannot be attributed to these components; a baseline with the same anchors and feature map but without attention and FPN (e.g., plain temporal convolutions) is needed. Without such an ablation, the paper's central claim that RapNet's relation-aware design is responsible for the competitive result is unsupported.
[Sec. 1] The fixed resizing of all videos to 128 snippets is a load-bearing preprocessing choice that is not analyzed. The manuscript gives no evidence that compressing arbitrarily long or short videos to 128 snippets preserves the boundary information that the proposal network and the boundary-adjustment stages rely on. A comparison with at least one other temporal length (e.g., 256 snippets) or a breakdown of recall by action duration would be necessary to rule out that this resizing degrades performance for a significant subset of actions.
[Secs. 3 and 4 (overall evaluation)] The paper provides no comparison with state-of-the-art methods on ActivityNet v1.3 validation (e.g., BSN, BMN, CTAP) and no error bars or number of runs. Consequently, the improvements attributed to boundary adjustment (0.74% and then 0.30% AUC in Table 2) could be within run-to-run variance, and the absolute values cannot be contextualized against published results. Reporting standard deviations over multiple training runs and a table of prior published AUC and AR@AN numbers is necessary to support the claimed significance of the increments.

minor comments (5)

[Abstract and Sec. 1] There are typos such as "totem-poral" in the abstract and "proposasl" in Section 1; the manuscript would benefit from a proofreading pass.
[Sec. 2, Eq. (1)] The loss notation is difficult to follow: the symbols T, M, N, and the index ranges (especially the T/2^i term) are not defined explicitly, and the relationship between the pyramid levels and the summation bounds is left implicit.
[Sec. 2] The sentence "In experiment, we find 12 anchors achieve the best performance" reports a selection made on the validation set without specifying the search range or the criterion; this is a free parameter that should be documented for reproducibility.
[Sec. 3] The typo "degisn" and the phrase "with frames actioness" obscure the description of the boundary-adjustment scheme; please clarify which features the PEM and watershed stages operate on.
[Sec. 4] The ensemble description does not specify how the two backbone outputs and two anchor configurations are fused (e.g., score averaging, proposal merging, NMS thresholds), which is needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RapNet's benchmark numbers come from an external challenge server and no prediction reduces by construction to a fitted input.

full rationale

The paper's derivation chain is empirical and externally anchored. Video features are extracted with a ResNet-C3D backbone fine-tuned on ActivityNet v1.3, proposals are generated from K-means-selected anchors with a supervised loss (objectness, regression, IoU) against ground-truth annotations, boundaries are adjusted using BSN's PEM, soft-NMS, and TAG's watershed algorithm, and final performance is measured on the ActivityNet validation set and the challenge testing server. No equation in the paper defines a predicted quantity as equal to a fitted parameter or to the model's own input, and no load-bearing claim is justified by a self-citation: the cited works (ResNet, C3D, FPN, self-attention, BSN, soft-NMS, TAG, YOLO) are all external prior art. The use of validation performance to select anchor counts and ensemble components is standard supervised model selection, not circular prediction. The absence of an ablation isolating the self-attention and FPN modules is a legitimate attribution/correctness concern, but it does not make the reported benchmark numbers circular, because those numbers are obtained from an independent evaluation protocol rather than derived from the method's assumptions. Overall, the paper shows no significant circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central empirical result depends on a set of hand-selected hyperparameters and input assumptions, none of which are derived or ablated. These parameters are standard for the method class, but they are not justified beyond validation performance. No new physical or ontological entities are introduced; RapNet is a composition of existing components.

free parameters (6)

Anchor count (K-means) = 12
Selected as best on validation; anchors are fitted to ActivityNet training action durations and affect proposal scales.
Temporal sequence length T' = 128
All videos resized to 128 snippets before proposal generation; assumes this preserves boundaries.
Negative screening threshold theta_iou = not specified
Controls the ignore strategy in confidence loss; no value or sensitivity analysis given.
Loss weights lambda_conf, lambda_c, lambda_w, lambda_iou = 0.2, 1, 1, 1
Hand-set weights for confidence, center, width, and IoU losses.
Number of pyramid generators = 6
Chosen architecture detail; no ablation reported.
Ensemble composition = ResNet-50 and ResNet-101 with anchor sets 12 and 18
Final ensemble combines two backbones and two anchor sets; no details on averaging or selection.

assumptions (3)

domain assumption Resizing all videos to a fixed 128-snippet representation preserves the temporal boundaries needed for action proposal generation.
Invoked in Sec. 1 when every video representation is fixed to 128x256; if boundary information is lost, proposals are unreliable.
domain assumption K-means derived anchors on the ActivityNet training set generalize to test videos.
Sec. 2 uses K-means to select anchors; this assumes the training distribution of action durations covers the test distribution.
domain assumption IoU-based one-to-one label assignment with a screening threshold is sufficient to supervise proposal confidence.
Sec. 2 defines positive and negative labels via highest IoU; this is a design choice not validated with ablations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Relation-Aware Pyramid Network (RapNet) for temporal action proposal." pith.science (2026). https://pith.science/paper/FK4FG2PL

@misc{pith2026190803448,
  author       = {Pith},
  title        = {Pith review of: Relation-Aware Pyramid Network (RapNet) for temporal action proposal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FK4FG2PL}},
  note         = {Machine review of arXiv:1908.03448}
}

read the original abstract

In this technical report, we describe our solution to temporal action proposal (task 1) in ActivityNet Challenge 2019. First, we fine-tune a ResNet-50-C3D CNN on ActivityNet v1.3 based on Kinetics pretrained model to extract snippet-level video representations and then we design a Relation-Aware Pyramid Network (RapNet) to generate temporal multiscale proposals with confidence score. After that, we employ a two-stage snippet-level boundary adjustment scheme to re-rank the order of generated proposals. Ensemble methods are also been used to improve the performance of our solution, which helps us achieve 2nd place.

Figures

Figures reproduced from arXiv: 1908.03448 by the authors.

**Figure 1.** The architecture of our action proposal generation network. The orange blocks and blue ones are enhanced by self-attention [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 7 canonical work pages

[1]

Bodla, B

N. Bodla, B. Singh, R. Chellappa, and L. S. Davis. Soft-nms--improving object detection with one line of code. In Proceedings of the IEEE International Conference on Computer Vision , pages 5561--5569, 2017

work page 2017
[2]

Carreira and A

J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299--6308, 2017

work page 2017
[3]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009

work page 2009
[4]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778, 2016

2016
[5]

T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 3--19, 2018

work page 2018
[6]

T.-Y. Lin, P. Doll \'a r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2117--2125, 2017

work page 2017
[7]

Redmon and A

J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018

arXiv 2018
[8]

D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 4489--4497, 2015

work page 2015

Show all 11 references

[9]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, . Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998--6008, 2017

2017
[10]

Xiong, Y

Y. Xiong, Y. Zhao, L. Wang, D. Lin, and X. Tang. A pursuit of temporal accuracy in general activity detection. arXiv preprint arXiv:1703.02716 , 2017

2017 arXiv
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Bodla, B

N. Bodla, B. Singh, R. Chellappa, and L. S. Davis. Soft-nms--improving object detection with one line of code. In Proceedings of the IEEE International Conference on Computer Vision , pages 5561--5569, 2017

work page 2017

[2] [2]

Carreira and A

J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299--6308, 2017

work page 2017

[3] [3]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009

work page 2009

[4] [4]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778, 2016

2016

[5] [5]

T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 3--19, 2018

work page 2018

[6] [6]

T.-Y. Lin, P. Doll \'a r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2117--2125, 2017

work page 2017

[7] [7]

Redmon and A

J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018

arXiv 2018

[8] [8]

D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 4489--4497, 2015

work page 2015