REVIEW 3 major objections 4 minor 40 references
Progressive Relation Learning for Group Activity Recognition
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Two reinforcement-learning agents progressively select the defining frames and relations, lifting group activity recognition accuracy above its graph-only baseline.
desk verdict A plausible RL-based progressive relation learning method for group activity recognition with modest gains; deserves a serious referee but needs seed-level validation before the numbers are trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the semantic relation graph (SRG), a graph network in which nodes are people, edges are relation vectors, and a global attribute carries the activity score; messages pass along edges for several iterations. Around it sit two reinforcement-learning agents trained with the A3C algorithm. The feature-distilling (FD) agent works in a discrete action space, keeping or replacing candidate frames. The relation-gating (RG) agent works in a continuous action space, producing a gate in [0,1] per edge, and its rewards combine the sign of the change in the L2,1 norm of the gate matrix with the sign of the change in the predicted probability of the ground-truth label. Alternate nine-stage training, freezing two components while updating the third, is what lets the graph and the two agents improve each other.
What would settle it
Train the same nine-stage procedure with the RG agent's gates replaced by uniform random gates in [0,1] (or with the sign rewards replaced by their raw magnitudes) and compare Volleyball MCA; if accuracy does not drop from 91.4%, the progressive relation-gating mechanism is not load-bearing.
Extended reading notes
Core claim
On its own terms, the paper claims that group activity recognition improves when the model explicitly represents each pairwise relation as a learnable vector in a semantic relation graph and then progressively refines both the input features and the graph. A feature-distilling agent selects which frames of the per-person spatiotemporal features to keep, while a relation-gating agent multiplies each edge by a continuous gate, with rewards that encourage structured row sparsity and an ascending probability for the true activity label. The three components are trained alternately in nine stages. The resulting PRL framework reaches 91.4% MCA on Volleyball, 1.3% above the best prior no-optical-flow method on Collective Activity, and its gain over the graph-only baseline (3.1% MCA) is larger than the sum of the gains from the two agents alone (2.7%), which the paper reads as evidence that the two agents reinforce each other.
Load-bearing premise
The load-bearing premise is that rewarding small improvements in the model's confidence and in the sparsity of the relation gates is a stable learning signal across all nine training stages, so the two agents genuinely improve rather than chase random fluctuations.
Editorial extensions
If this is right
- Explicitly representing relations as edge vectors in a graph network outperforms passing messages implicitly: the SRG alone beats the message-passing baseline on Volleyball (88.3 vs 87.9 MCA).
- The RL-based agents outperform their attention-based counterparts: SRG+FD beats SRG+temporal attention by 0.9 MCA and SRG+RG beats SRG+relation attention by 1.1 MCA on Volleyball.
- Progressive refinement of frames and relations is complementary: the full PRL improves over the SRG by 3.1 MCA, more than the sum of the two individual gains, implying mutual reinforcement through alternate training.
- Without any optical flow input, the method is on par with or ahead of several prior systems that use optical flow, and the learned gate values concentrate on participants whose actions define the activity, as shown in the visualized graphs.
Reading between the lines
- A consequence the authors do not draw: the gate matrix itself is a per-relation importance map, so the framework can be read as an interpretability tool for why a group activity was predicted.
- A testable extension: feed the distilled frames and gated relations into an online recognition loop, using the FD queue as a temporal buffer, to see if progressive refinement helps before the full clip is seen.
- If the sign rewards are the fragile part, a natural variant replaces them with scaled probability-difference rewards; comparing training stability would separate the contribution of progressive refinement from the reward shaping.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes Progressive Relation Learning (PRL) for group activity recognition. It constructs a Semantic Relation Graph (SRG) over person nodes and relation edges, then introduces two reinforcement-learning agents: a feature-distilling (FD) agent that selects informative frames from the low-level spatiotemporal features, and a relation-gating (RG) agent that multiplies each relation edge by a continuous gate. The graph and agents are trained alternately over nine stages. Experiments on the Volleyball and Collective Activity datasets report MCA/MPCA improvements over the authors' own SRG baseline and results on par with or better than prior methods. The main empirical claims are that PRL improves MCA by 3.1% over SRG on Volleyball and MPCA by 1.3% over SPA+KD on CAD.
Significance. If the empirical results are stable, the paper makes a useful contribution: it demonstrates that explicit relation-level gating learned by reinforcement learning can improve group activity recognition over a graph-only baseline, and the ablation decomposition (SRG vs. SRG+FD vs. SRG+RG vs. PRL) is informative. The paper also provides qualitative visualizations of the learned relation graphs. However, the central empirical claims rest on single-run results and on a sign-based reward whose stability is not demonstrated; these issues must be addressed before the contribution can be considered established.
major comments (3)
- [§3.3, Eqs. (7)-(10)] The rewards r_sparse and r_ascend are sign functions of single-step differences in the L2,1 norm and in the groundtruth-label probability p_c. Because the RG agent emits one gate at a time (Sec. 3.3, 'Relation-gating Agent'), the change in p_c after a single edge update is typically small relative to classifier stochasticity, and the sign transform can turn this small delta into an essentially random ±1 reward. This is load-bearing because the reported 3.1% MCA gain of PRL over SRG (Table 1) is attributed to the progressive training loop of Sec. 3.5. The manuscript reports no learning curves, reward traces, gate-value statistics, or sensitivity analysis for the reward weights, so the stability and informativeness of this reward are not established. I request such diagnostics, and also a concrete test of the sign nonlinearity, e.g., ablating it to a saturating linear reward or reporting the variance of r_ascend across steps for the trained agents.
- [§4.4, Tables 1-2] The headline improvements (PRL over SRG by 3.1% MCA on Volleyball; PRL over SPA+KD by 1.3% MPCA on CAD) are presented without error bars, multiple seeds, or a description of how the reported checkpoint was selected. Given the stochasticity of A3C training (16 asynchronous workers, sampled actions, Eq. 12), the differences between variants, which are often around 1%, may be within run-to-run variance. The central empirical claim therefore needs support from means and standard deviations over at least three seeds, or an explicit statement of the evaluation protocol and number of runs.
- [§3.5, Alternate Training] In the nine-stage alternate training, the reward for each agent is generated by the currently frozen SRG and FD features, but the agent's policy is later used with a retrained SRG. The paper does not report the evolution of training rewards or validation accuracy across the nine stages, nor does it show that the alternating procedure converges rather than oscillating or overfitting to the frozen component at each stage. Please report per-stage validation accuracy (at least after stages 3, 6, and 9) and the reward curves for both agents, and state the stopping criterion used for the 2-hour agent training phases.
minor comments (4)
- [§3.3] The text says 'we normalize the values of gates before gating operation, i.e., gij = gji = (gij+gji)/2'; this is symmetrization, not normalization, and the terminology should be corrected.
- [§1] There are several typos, e.g., 'strucural RNNs' and 'the course individual level' should read 'structural RNNs' and 'the coarse individual level'.
- [§4.2] The implementation details state hyperparameter values (tau_max = 5, Omega = 15/20, Td = 5) but do not report any sensitivity analysis; a short discussion or reference to supplementary experiments would help the reader judge robustness.
- [§4.4] The sentence 'the PRL eventually improves 3.1% over the original SRG, which is even larger than the sum of increments from the two agents, 2.7%' is presented as evidence of mutual boosting; however, incremental gains in an ablation are not additive by construction, so this argument should be softened or supported by explicitly comparing the gain of PRL over SRG+FD and SRG+RG.
Circularity Check
No significant circularity: the method is trained and evaluated on external benchmarks, and the self-citations are not load-bearing.
full rationale
The paper makes no mathematical derivation in which an output quantity is defined in terms of the quantity it is said to predict. The central claim, that progressive relation learning (PRL) improves group activity recognition, is supported by accuracy measurements on the Volleyball and CAD benchmarks (Tables 1 and 2) relative to the authors' own SRG baseline and to prior published methods. The RG and FD agents are trained with A3C using rewards defined in Eqs. 7-10 and 13; although the ascending reward in Eq. 8 uses the predicted probability of the groundtruth label, this is a supervised training signal, not a fitted parameter renamed as a prediction. The reported test-set accuracies are not optimized against the test set and are therefore not circular. The only self-citations, Refs. [12] and [13], appear in the Related Works section as examples of graph neural network usage for skeleton-based action recognition; they are not invoked to justify the present method's correctness or to forbid alternatives. The skeptical concern about the sign-based reward being noisy or prone to reward hacking is a robustness/reproducibility issue, not a circularity issue. Accordingly, the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Td (number of frames kept by FD agent) =
5
- m (number of graph propagation iterations) =
3
- tau_max (reinforcement updating interval) =
5 for both agents
- Omega (stimulation and punishment strength in rshift) =
15 for RG, 20 for FD
- Reward weights for rsparse, rascend, rshift =
1 each
assumptions (5)
- standard math A3C algorithm provides stable policy-gradient updates for both discrete and continuous action agents.
- domain assumption The group activity label in CAD can be treated as the majority action label of individuals.
- domain assumption Person bounding boxes from the Dlib tracker are accurate and available.
- ad hoc to paper The reward design consisting of the sum of rsparse, rascend, and rshift provides a valid learning signal for relation gating.
- domain assumption Pre-extracted VGG16 and LSTM features, frozen after preparation, are sufficient for the semantic relation graph.
Cite this review
Pith. "Pith review of Progressive Relation Learning for Group Activity Recognition." pith.science (2026). https://pith.science/paper/GISZOQXR
@misc{pith2026190802948,
author = {Pith},
title = {Pith review of: Progressive Relation Learning for Group Activity Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/GISZOQXR}},
note = {Machine review of arXiv:1908.02948}
}
read the original abstract
Group activities usually involve spatiotemporal dynamics among many interactive individuals, while only a few participants at several key frames essentially define the activity. Therefore, effectively modeling the group-relevant and suppressing the irrelevant actions (and interactions) are vital for group activity recognition. In this paper, we propose a novel method based on deep reinforcement learning to progressively refine the low-level features and high-level relations of group activities. Firstly, we construct a semantic relation graph (SRG) to explicitly model the relations among persons. Then, two agents adopting policy according to two Markov decision processes are applied to progressively refine the SRG. Specifically, one feature-distilling (FD) agent in the discrete action space refines the low-level spatio-temporal features by distilling the most informative frames. Another relation-gating (RG) agent in continuous action space adjusts the high-level semantic graph to pay more attention to group-relevant relations. The SRG, FD agent, and RG agent are optimized alternately to mutually boost the performance of each other. Extensive experiments on two widely used benchmarks demonstrate the effectiveness and superiority of the proposed approach.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Convolutional relational machine for group activity recognition
Sina Mokhtarzadeh Azar, Mina Ghadimi Atigh, Ahmad Nickabadi, and Alexandre Alahi. Convolutional relational machine for group activity recognition. In CVPR, pages 7892–7901, 2019. 7, 8
work page 2019
-
[2]
Bagautdinov, Alexandre Alahi, Franc ¸ois Fleuret, Pascal Fua, and Silvio Savarese
Timur M. Bagautdinov, Alexandre Alahi, Franc ¸ois Fleuret, Pascal Fua, and Silvio Savarese. Social scene understand- ing: End-to-end multi-person action localization and collec- tive activity recognition. In CVPR, pages 3425–3434, 2017. 7
work page 2017
-
[3]
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vin´ıcius Flores Zambaldi, Mateusz Ma- linowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, C ¸ aglar G¨ulc ¸ehre, Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicola...
arXiv 2018
-
[4]
Structural recurrent neural network (SRNN) for group activity analysis
Sovan Biswas and Juergen Gall. Structural recurrent neural network (SRNN) for group activity analysis. InWACV, pages 1625–1632, 2018. 1, 7
work page 2018
-
[5]
Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Process. Mag. , 34(4):18–42, 2017. 2
work page 2017
-
[6]
Quo vadis, action recognition? a new model and the kinetics dataset
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299–6308, 2017. 7
2017
-
[7]
A unified framework for multi-target tracking and collective activity recognition
Wongun Choi and Silvio Savarese. A unified framework for multi-target tracking and collective activity recognition. In ECCV, pages 215–230. Springer, 2012. 1
work page 2012
-
[8]
Wongun Choi, Khuram Shahid, and Silvio Savarese. What are they doing?: Collective activity classification using spatio-temporal relationship among people. In ICCV Work- shops, pages 1282–1289. IEEE, 2009. 6
work page 2009
Show all 40 references
-
[9]
Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition
Zhiwei Deng, Arash Vahdat, Hexiang Hu, and Greg Mori. Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition. In CVPR, pages 4772–4781, 2016. 1
2016
-
[10]
Schoenholz, Patrick F
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263–1272, 2017. 2
2017
-
[11]
Visual recognition by counting instances: A multi- instance cardinality potential kernel
Hossein Hajimirsadeghi, Wang Yan, Arash Vahdat, and Greg Mori. Visual recognition by counting instances: A multi- instance cardinality potential kernel. In CVPR, pages 2596– 2605, 2015. 8
2015
-
[12]
Joint learning in the spatio- temporal and frequency domains for skeleton-based action recognition
Guyue Hu, Bo Cui, and Shan Yu. Joint learning in the spatio- temporal and frequency domains for skeleton-based action recognition. IEEE Transactions on Multimedia, 2019. 2
2019
-
[13]
Skeleton-based action recognition with synchronous local and non-local spatio- temporal learning and frequency attention
Guyue Hu, Bo Cui, and Shan Yu. Skeleton-based action recognition with synchronous local and non-local spatio- temporal learning and frequency attention. In 2019 IEEE International Conference on Multimedia and Expo (ICME) , pages 1216–1221, 2019. 2
2019
-
[14]
Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori
Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierarchical deep temporal model for group activity recognition. In CVPR, pages 1971– 1980, 2016. 6, 7, 8
1971
-
[15]
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10:1755–1758, 2009. 2
2009
-
[16]
Konda and John N
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algo- rithms. In NIPS, pages 1008–1014, 1999. 2, 4, 5
1999
-
[17]
Robi- novitch, and Greg Mori
Tian Lan, Yang Wang, Weilong Yang, Stephen N. Robi- novitch, and Greg Mori. Discriminative latent models for recognizing contextual group activities. IEEE Trans. Pattern Anal. Mach. Intell., 34(8):1549–1562, 2012. 1
2012
-
[18]
A2- RL: aesthetics aware reinforcement learning for image crop- ping
Debang Li, Huikai Wu, Junge Zhang, and Kaiqi Huang. A2- RL: aesthetics aware reinforcement learning for image crop- ping. In CVPR, pages 8193–8201, 2018. 2
2018
-
[19]
SBGAR: semantics based group activity recognition
Xin Li and Mooi Choo Chuah. SBGAR: semantics based group activity recognition. In ICCV, pages 2895–2904,
-
[20]
Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu
V olodymyr Mnih, Adri`a Puigdom`enech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016. 2, 5
2016
-
[21]
Riedmiller
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. 2
2013 arXiv
-
[22]
stagnet: An attentive semantic RNN for group activity recognition
Mengshi Qi, Jie Qin, Annan Li, Yunhong Wang, Jiebo Luo, and Luc Van Gool. stagnet: An attentive semantic RNN for group activity recognition. In ECCV, pages 104–120, 2018. 1, 2, 6, 7, 8
2018
-
[23]
Bernstein, Alexander C
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition chal- lenge. IJCV, 115(3):211–252, 2015. 6
2015
-
[24]
Riedmiller, Raia Hadsell, and Peter Battaglia
Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Sprin- genberg, Josh Merel, Martin A. Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics en- gines for inference and control. In ICML, pages 4467–4476,
-
[25]
CERN: confidence-energy recurrent network for group activity recognition
Tianmin Shu, Sinisa Todorovic, and Song-Chun Zhu. CERN: confidence-energy recurrent network for group activity recognition. In CVPR, pages 4255–4263, 2017. 6, 7, 8
2017
-
[26]
Joint inference of groups, events and human roles in aerial videos
Tianmin Shu, Dan Xie, Brandon Rothrock, Sinisa Todorovic, and Song-Chun Zhu. Joint inference of groups, events and human roles in aerial videos. In CVPR, pages 4576–4584,
-
[27]
Dynamic edge- conditioned filters in convolutional neural networks on graphs
Martin Simonovsky and Nikos Komodakis. Dynamic edge- conditioned filters in convolutional neural networks on graphs. In CVPR, pages 29–38, 2017. 2
2017
-
[28]
Very deep convo- lutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. InICLR,
-
[29]
Sutton, David A
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In NIPS, pages 1057–1063, 1999. 2
1999
-
[30]
Deep progressive reinforcement learning for skeleton-based action recognition
Yansong Tang, Yi Tian, Jiwen Lu, Peiyang Li, and Jie Zhou. Deep progressive reinforcement learning for skeleton-based action recognition. In CVPR, pages 5323–5332, 2018. 4
2018
-
[31]
Mining semantics-preserving attention for group activity recognition
Yansong Tang, Zian Wang, Peiyang Li, Jiwen Lu, Ming Yang, and Jie Zhou. Mining semantics-preserving attention for group activity recognition. In ACM MM, pages 1283–
-
[32]
Recurrent modeling of interaction context for collective activity recog- nition
Minsi Wang, Bingbing Ni, and Xiaokang Yang. Recurrent modeling of interaction context for collective activity recog- nition. In CVPR, 2017. 1, 6
2017
-
[33]
Learning actor relation graphs for group activity recognition
Jianchao Wu, Limin Wang, Li Wang, Jie Guo, and Gang- shan Wu. Learning actor relation graphs for group activity recognition. In CVPR, pages 9964–9974, 2019. 7
2019
-
[34]
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596,
1901 arXiv
-
[35]
How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018. 2
2018 arXiv
-
[36]
Participation-contributed temporal dynamic model for group activity recognition
Rui Yan, Jinhui Tang, Xiangbo Shu, Zechao Li, and Qi Tian. Participation-contributed temporal dynamic model for group activity recognition. In ACM MM, pages 1292–1300. ACM,
-
[37]
Spatial tempo- ral graph convolutional networks for skeleton-based action recognition
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial tempo- ral graph convolutional networks for skeleton-based action recognition. In AAAI, pages 7444–7452, 2018. 2
2018
-
[38]
Visual semantic navigation using scene priors
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. arXiv preprint arXiv:1810.06543, 2018. 2
2018 arXiv
-
[39]
Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI, pages 3634–3640, 2018. 2
2018
-
[1291]
1, 6, 7, 8
ACM, 2018. 1, 6, 7, 8
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.