REVIEW 5 major objections 5 minor 1 cited by
FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A frequency-split attention design cuts a skeleton-action transformer's parameters by 40% while losing at most 0.8 accuracy points.
desk verdict A derivative but useful lightweight skeleton transformer with a genuinely new high-low frequency operator, whose headline accuracy claim is not yet statistically supported because the tuning and evaluation share the same split and a single seed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's main mechanism is a high-low frequency operator on DCT coefficients. For a skeleton sequence, each joint trajectory is transformed with the Discrete Cosine Transform; a division coefficient N marks the first N coefficients as low-frequency and the rest as high-frequency. The operator scales low-frequency coefficients by l (set to 0.2) and high-frequency coefficients by h (set to 1.2), then applies the inverse DCT. The modulated trajectories feed two dedicated attention blocks, a High-Frequency Attention Block (HFAB) and a Low-Frequency Attention Block (LFAB), which are combined with a Spatial Attention Block (SAB) and a Temporal Attention Block (TAB) through mixed self-attention and cross-attention maps. This design, with only two channel-split units instead of four, carries the claimed parameter reduction from 2.0M to 1.2M.
What would settle it
Run the released model and FreqMixFormer for five seeds each on NTU-60 X-Sub and NTU-120 X-Sub, then recompute the parameter-accuracy trade-off; if the 0.8% gap is smaller than the seed spread, or if a different (N, h, l) beats (13, 1.2, 0.2) on NTU-120 by more than 0.5%, the claimed stable efficiency-accuracy balance fails.
Extended reading notes
Core claim
The central claim is that the previous FreqMixFormer's accuracy at 91.5% on NTU-60 X-Sub can be preserved at 90.7% with a model of 1.2M parameters, a 40% parameter cut, by redesigning rather than merely shrinking. The discovery, as the authors present it, is that the expensive part of frequency-aware attention was not the DCT itself but the redundant mixed-attention blocks and the single frequency operator that treated all coefficients alike. Splitting the DCT coefficients at N=13 into a low-frequency band (scaled down by l=0.2) and a high-frequency band (scaled up by h=1.2), and reducing the block count to 2 HFAB, 2 LFAB, 1 SAB, and 1 TAB, keeps the model within 0.8 accuracy points of the original across all three benchmarks. The same high-low operator also lets the 4-stream ensemble reach 92.9% X-View, 90.0% X-Set, and 97.0% on NW-UCLA, matching or beating the original FreqMixFormer's ensemble with fewer parameters.
Load-bearing premise
The three tuned coefficients (N=13, h=1.2, l=0.2), chosen by a single run on NTU-60 X-Sub, are assumed to remain optimal on every other dataset, and the reported tenth-of-a-percent accuracy differences are assumed to be real signal rather than run-to-run noise.
Editorial extensions
If this is right
- Cutting the network from 2.0M to 1.2M parameters means the model fits in roughly 4.8 MB of storage in single precision, a direct arithmetic consequence of the reported parameter count.
- The 4-stream ensemble stays within 0.4 points of the original FreqMixFormer on every NTU-120 and NW-UCLA setting while using 1.2M parameters, as shown in Table II.
- The appendix ablation credits the combined HFAB, LFAB, TAB stack with a 2.2-point gain over the stripped baseline, indicating the three attention blocks together, not any single one, carry the accuracy.
- The Table III search shows the chosen N=13 and (h,l)=(1.2,0.2) sit near the middle of a flat accuracy plateau, so nearby hyperparameter choices cost no more than 0.5 to 0.7 points.
- On confusing actions in the hard set, the model surpasses Hyperformer and SkeMixFormer with fewer parameters, per the appendix, suggesting the efficiency gain is not bought by losing fine-grained motion detail.
Reading between the lines
- The design suggests high-frequency DCT coefficients, not low-frequency shape, are where subtle actions are encoded; a testable extension is to feed only the high-frequency band to a standard skeleton model and measure the accuracy contribution.
- The same N/h/l frequency-splitting operator could be dropped into other sequence models beyond transformers, such as temporal convolutions or recurrent networks, providing a cheap way to emphasize fine-grained motion.
- Neither the paper nor the ablation isolates how much of the 40% parameter cut comes from reducing block repetition versus the new operator; a controlled thinning of the original FreqMixFormer would determine whether the operator is necessary for the efficiency gain.
- Because the reported numbers are single-run, a multi-seed re-run with released code would turn the 0.8% trade-off into a confidence interval; until then, the trade-off is a point estimate, not a bound.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FreqMixFormerV2, a lightweight variant of the frequency-aware mixed transformer for skeleton-based action recognition. The main architectural changes are: (i) reducing the number of attention blocks from 15 to 6, (ii) splitting the frequency operator into separate high-frequency and low-frequency operators controlled by coefficients h and ℓ and a partition parameter N, and (iii) restricting the input channel split from four units to two. The authors claim that the resulting 1.2M-parameter model achieves accuracy comparable to the 2.0M-parameter FreqMixFormer, with only a 0.8% drop on NTU-60 X-Sub, and they report results on NTU-60, NTU-120, and NW-UCLA. The paper includes comparisons with prior methods, an ablation study, and a per-action-type analysis.
Significance. If the results are reliable, the paper would provide a practically useful efficiency-accuracy trade-off for transformer-based skeleton action recognition: a 1.2M-parameter model that stays within about 0.5–0.8% of a 2.0M model across several benchmarks. The paper is also transparent about its goal of reducing deployment costs and includes experiments on three standard datasets, an efficiency-accuracy scatter plot, and a confusion-action analysis. However, the current evidence is weakened by hyperparameter selection on the test split, single-run results without variance estimates, and multiple internal inconsistencies in the ablation and reported numbers. The central comparative claim is therefore not yet statistically supported.
major comments (5)
- [Section III-C, Table III] The hyperparameters N, h, and ℓ are tuned directly on the NTU-60 X-Sub test accuracy, and the same split is used for the headline comparison in Table II. No validation split is held out. Since the entire search range covers only 0.7% (90.0 to 90.7) and every number in Table III is from a single run, the reported 90.7% may be an overfit maximum. The claim that FreqMixFormerV2 is within 0.8% of FreqMixFormer is only meaningful if 90.7% is a stable, selection-bias-free estimate; the authors should either select hyperparameters on a validation split or report multi-seed means with confidence intervals for the final configuration and for the comparisons.
- [Appendix, Table IV] The ablation table does not match the prose. The row with 89.2% has TAB=True and HFAB=False, yet the text credits HFAB for the 0.7% improvement; the row with 90.1% has HFAB=True and LFAB=False, but the text credits LFAB for the 1.6% improvement; the row with 90.2% has LFAB=True. Furthermore, the combined HFAB+LFAB row (89.4%) is worse than either block alone (90.1% and 90.2%), which contradicts the claimed additive benefit of the two frequency blocks. This internal inconsistency prevents the ablation from supporting the architectural design.
- [Section II-C.2, Eq. (15)] The frequency partition coefficient N is defined as N ∈ (1,J), where J is the number of joints, but the DCT in Eq. (15) is applied along the frame dimension F of each joint trajectory, producing F frequency coefficients. The search over N in Table III (N=1..17, with 25 joints in NTU-60) is therefore over the wrong dimension. There is no explanation of why the number of joints should bound the DCT coefficient partition, and the appendix's DCT formulation is inconsistent with the architecture text. This is a core correctness issue in the definition of the proposed high-low-frequency operator.
- [Section III-B] The text states that FreqMixFormerV2 achieves "90.7% on the X-Sub setting and 92.9% on the X-View setting" with 1.2M parameters. This contradicts Table II, which reports 90.7% X-Sub and 95.4% X-View for the joint modality, and 92.9% X-Sub and 96.9% X-View for the 4-stream ensemble. The 92.9% number is actually the 4-stream X-Sub result, not the joint X-View result. The reported comparison with FreqMixFormer is therefore misstated in the main text, and the efficiency claim attached to the 92.9% figure is misleading.
- [Section III-C.2, Table III] The search over h and ℓ is not a search over independent combinations: the rows always satisfy h = 1+ℓ (0.1/1.1, 0.2/1.2, ..., 0.9/1.9). The text calls this "different combinations of the high-frequency operator h and low-frequency operator ℓ," but only one degree of freedom is actually varied. Consequently, the paper provides no evidence that independent tuning of h and ℓ is beneficial, and the statement that "the combination of ℓ=0.2 and h=1.2 achieves the highest accuracy" is not supported by a joint grid search.
minor comments (5)
- [Abstract] The model name is misspelled as "FreqMixForemrV2" in the abstract; it should be "FreqMixFormerV2."
- [Section II-A, Table I] The text says the simplified architecture uses "2 blocks for each module vs. 4 blocks for each module," but Table I lists FreqMixFormer as having 7 FAB, 7 SAB, and 1 TAB (15 total), and FreqMixFormerV2 as having 2 HFAB, 2 LFAB, 1 SAB, and 1 TAB (6 total). The numbers do not match the stated 4-block baseline, so the comparison is unclear.
- [Section II-C.2] The constraint h ∈ (1, 1+ℓ) is written as if h and ℓ are independent, but the experiments only consider h = 1+ℓ. Please clarify whether h is intended to be a separate tunable parameter or a derived quantity.
- [Abstract, Section I] The abstract claims the model is "outperforming state-of-the-art methods," but Table II shows that FreqMixFormerV2 is below several existing methods in accuracy (e.g., J modality: 90.7 vs 91.5 for FreqMixFormer and 93.0 for HD-GCN; 4S: 92.9 vs 93.4 for FreqMixFormer). The defensible claim is a favorable efficiency-accuracy trade-off, not outright state-of-the-art accuracy.
- [Fig. 1, Table II] The scatter plot does not include error bars or markers that distinguish single-run results, and the label "state-of-the-art accuracy" is not justified by Table II. Also, the parameter counts for 4-stream results are reported as per-modality values; this should be stated explicitly in the table caption.
Circularity Check
Headline NTU-60 X-Sub accuracy is the maximum of a test-set hyperparameter grid search, so the '0.8% reduction' claim is not an independent prediction.
-
fitted input called prediction
[Section III-C (Table III), Table II, and Abstract]
"Search for the best partition N ... As shown in Table. III, partition N = 13 achieves the best accuracy of 90.7% on NTU-60 X-Sub. ... The results show that the combination of ℓ = 0.2 and h = 1.2 achieves the highest accuracy of 90.7%. ... FreqMixFormerV2 achieves comparable performance with only a 0.8% reduction in accuracy, while using just 60% of the parameter size of the original model."
The hyperparameters N, h, and ℓ are selected by maximizing accuracy on the NTU-60 X-Sub split (Table III), and the same split's post-selection accuracy (90.7%) is then reported in Table II and used in the abstract to claim 'comparable performance with only a 0.8% reduction in accuracy' relative to FreqMixFormer's 91.5%. Because 90.7% is, by construction, the maximum over the searched grid on exactly the test split used for evaluation, the headline gap of 0.8% is the value being optimized, not an independent held-out estimate. The 'prediction' of comparable performance on NTU-60 X-Sub is therefore statistically forced by the selection procedure.
full rationale
I found one concrete selection-on-test-set circularity, scored as partial. The paper's equations (1)-(14) are not restatements of their inputs: the high-low frequency split (Eqs. 5-9) and mixed spatial attention (Eqs. 2-4) follow from defined DCT and attention operations, and the method is not a renaming of a known result. Self-citations to [18]/[19] provide the inherited architecture and the FreqMixFormer comparison baseline; that baseline (91.5% on X-Sub) comes from the authors' prior work, but the V2 accuracy is measured here, so the comparison is not a self-justifying theorem. The ablation section has internal label/table mismatches (the 89.2 row is described as HFAB while the table shows TAB, and the combined HFAB+LFAB row 89.4 is worse than either alone), which undermines the incremental contribution claim but is a consistency problem rather than circularity. Overall, the derivation is not fully circular, but the headline accuracy for NTU-60 X-Sub is the optimized maximum of a grid search on the same test split, so the central efficiency-accuracy claim partially reduces to its own selection criterion.
Assumptions & free parameters
free parameters (5)
- Frequency partition N =
13
- High-frequency operator h =
1.2
- Low-frequency operator l =
0.2
- Block counts =
2 HFAB, 2 LFAB, 1 SAB, 1 TAB
- Input channel split =
2
assumptions (3)
- domain assumption DCT coefficient ordering: the first N coefficients are the low-frequency part and the remaining J-N are high-frequency.
- domain assumption DCT/IDCT round-trip and the frequency operators preserve the information needed for action classification after pooling and linear projections.
- domain assumption Standard NTU and NW-UCLA evaluation protocols and the multi-stream fusion scheme from prior work are correct and comparable across methods.
Cite this review
Pith. "Pith review of FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition." pith.science (2026). https://pith.science/paper/NVOFJ3EN
@misc{pith2026241220621,
author = {Pith},
title = {Pith review of: FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/NVOFJ3EN}},
note = {Machine review of arXiv:2412.20621}
}
read the original abstract
Transformer-based human skeleton action recognition has been developed for years. However, the complexity and high parameter count demands of these models hinder their practical applications, especially in resource-constrained environments. In this work, we propose FreqMixForemrV2, which was built upon the Frequency-aware Mixed Transformer (FreqMixFormer) for identifying subtle and discriminative actions with pioneered frequency-domain analysis. We design a lightweight architecture that maintains robust performance while significantly reducing the model complexity. This is achieved through a redesigned frequency operator that optimizes high-frequency and low-frequency parameter adjustments, and a simplified frequency-aware attention module. These improvements result in a substantial reduction in model parameters, enabling efficient deployment with only a minimal sacrifice in accuracy. Comprehensive evaluations of standard datasets (NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets) demonstrate that the proposed model achieves a superior balance between efficiency and accuracy, outperforming state-of-the-art methods with only 60% of the parameters.
Figures
Forward citations
Cited by 1 Pith paper
-
UniSTFormer: Unified Spatio-Temporal Lightweight Transformer for Efficient Skeleton-Based Action Recognition
UniSTFormer merges spatial and temporal attention for skeleton-based action recognition, achieving 91.0% on NTU-60 with 0.5M parameters and 1.1 GFLOPs.
Reference graph
Works this paper leans on
-
[1]
R. Bai, M. Li, B. Meng, F. Li, M. Jiang, J. Ren, and D. Sun. Hierarchi- cal graph convolutional skeleton transformer for action recognition. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 01–06. IEEE, 2022
work page 2022
-
[2]
Y . Chen, Z. Zhang, C. Yuan, B. Li, Y . Deng, and W. Hu. Channel- wise topology refinement graph convolution for skeleton-based action recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 13359–13368, 2021
work page 2021
-
[3]
Z. Chen, S. Li, B. Yang, Q. Li, and H. Liu. Multi-scale spatial temporal graph convolutional network for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 1113–1122, 2021
work page 2021
-
[4]
H.-g. Chi, M. H. Ha, S. Chi, S. W. Lee, Q. Huang, and K. Ramani. Infogcn: Representation learning for human skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 20186–20196, 2022
work page 2022
-
[5]
Z. Gao, P. Wang, P. Lv, X. Jiang, Q. Liu, P. Wang, M. Xu, and W. Li. Focal and global spatial-temporal transformer for skeleton- based action recognition. In Proceedings of the Asian Conference on Computer Vision, pages 382–398, 2022
work page 2022
-
[6]
J. Lee, M. Lee, D. Lee, and S. Lee. Hierarchically decomposed graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10444–10453, October 2023
work page 2023
-
[7]
H. Liu, Y . Liu, Y . Chen, C. Yuan, B. Li, and W. Hu. Transkeleton: Hierarchical spatial-temporal transformer for skeleton-based action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2023
work page 2023
-
[8]
J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y . Duan, and A. C. Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, 42(10):2684–2701, 2019
work page 2019
Show all 23 references
-
[9]
Z. Liu, H. Zhang, Z. Chen, Z. Wang, and W. Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 143–152, 2020
2020
-
[10]
Paszke, S
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019
2019
-
[11]
Plizzari, M
C. Plizzari, M. Cannici, and M. Matteucci. Skeleton-based action recognition via spatial and temporal transformer networks. Computer Vision and Image Understanding , 208:103219, 2021
2021
-
[12]
Shahroudy, J
A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1010–1019, 2016
2016
-
[13]
L. Shi, Y . Zhang, J. Cheng, and H. Lu. Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition. In Proceedings of the Asian conference on computer vision , 2020
2020
-
[14]
Y .-F. Song, Z. Zhang, C. Shan, and L. Wang. Constructing stronger and faster baselines for skeleton-based action recognition. IEEE transac- tions on pattern analysis and machine intelligence , 45(2):1474–1488, 2022
2022
-
[15]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017
2017
-
[16]
J. Wang, X. Nie, Y . Xia, Y . Wu, and S.-C. Zhu. Cross-view action modeling, learning and recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2649– 2656, 2014
2014
-
[17]
W. Wu, Y . Hua, C. Zheng, S. Wu, C. Chen, and A. Lu. Skeletonmae: Spatial-temporal masked autoencoders for self-supervised skeleton ac- tion recognition. In 2023 IEEE international conference on multimedia and expo workshops (ICMEW) , pages 224–229. IEEE, 2023
2023
-
[18]
W. Wu, C. Zheng, Z. Yang, C. Chen, S. Das, and A. Lu. Frequency guidance matters: Skeletal action recognition by frequency-aware mixed transformer. ACM Multimedia 2024 , 2024
2024
-
[19]
W. Xin, Q. Miao, Y . Liu, R. Liu, C.-M. Pun, and C. Shi. Skeleton mixformer: Multivariate topology representation for skeleton-based action recognition. In Proceedings of the 31st ACM International Conference on Multimedia , pages 2211–2220, 2023
2023
-
[20]
Zhang, B
Y . Zhang, B. Wu, W. Li, L. Duan, and C. Gan. Stst: Spatial-temporal specialized transformer for skeleton-based action recognition. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3229–3237, 2021
2021
-
[21]
H. Zhou, Q. Liu, and Y . Wang. Learning discriminative represen- tations for skeleton based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10608–10617, 2023
2023
-
[22]
Zhou, Z.-Q
Y . Zhou, Z.-Q. Cheng, C. Li, Y . Fang, Y . Geng, X. Xie, and M. Keuper. Hypergraph transformer for skeleton-based action recognition. arXiv preprint arXiv:2211.09590, 2022
2022 arXiv
-
[23]
Y . Zhou, X. Yan, Z.-Q. Cheng, Y . Yan, Q. Dai, and X.-S. Hua. Blockgcn: Redefine topology awareness for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2049–2058, 2024. APPENDIX Experiment Setting...
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.