REVIEW 3 major objections 4 minor 8 cited by
MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read MM-Gesture claims 73.213% top-1 on iMiGUE by fusing six modalities.
desk verdict Credible competition result with an externally verified first place, but the SOTA claim needs a broader comparison; the ensemble/transfer-learning recipe is a solid engineering contribution, not a scientific breakthrough. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a probability-level weighted ensemble (Eq. 7) that combines outputs from six modality-specific models. Skeleton joints and limbs are converted into Gaussian heatmap volumes and processed by PoseConv3D, with RGB fused through paired training losses. RGB, Taylor-series, optical-flow, and depth videos are each encoded by Video Swin Transformer, and the RGB branch is pretrained on MA-52 before fine-tuning on iMiGUE. The weights $w_i$ are set empirically from validation performance, which is what allows the final accuracy of 73.213%.
What would settle it
Retrain the six modality models, tune the ensemble weights on the validation set, and evaluate on the test set; if the RGB* model alone, with a weight of 1 for that modality and 0 for the others, matches or exceeds 73.213% on the test set, then the claimed benefit of the full multimodal ensemble is falsified.
Extended reading notes
Core claim
The paper's central claim is that a six-modality ensemble, with each modality-specific network trained separately and combined at the probability level, outperforms every prior micro-gesture method on the iMiGUE dataset. The authors report that a top-1 accuracy of 73.213% is achieved, ranking first in the 3rd MiGA Challenge at IJCAI 2025. The ablation results show a consistent incremental gain from adding modalities: skeleton plus RGB reaches 71.416%, adding the Taylor modality gives 72.096%, optical flow gives 72.227%, and depth gives 72.644%, with the final optimized weighting reaching 73.213%. Transfer learning from the MA-52 dataset improves the RGB branch, and the final ensemble weights are chosen empirically by validation performance.
Load-bearing premise
The load-bearing premise is that the ensemble weights $w_i$ tuned on the 777-sample validation split generalize to the 4,562-sample test split; if that premise fails, the reported 73.213% accuracy is not a reliable estimate of true performance.
Editorial extensions
If this is right
- If MM-Gesture is correct, fusing skeleton, RGB, Taylor, optical flow, and depth yields 73.213% top-1 accuracy on iMiGUE, about 3 points above the previous best challenge entry.
- Each added modality contributes a small but consistent gain, which suggests that the six modalities carry complementary information rather than redundant cues.
- Pretraining the RGB branch on the MA-52 dataset improves RGB-only accuracy, making transfer learning from a larger micro-action dataset a reusable ingredient.
- The winning result comes from a simple late-fusion weighted ensemble with validation-tuned weights, not from end-to-end joint training of all modalities.
- The ranking outcome implies that this combination of backbone architectures, modality extractors, and ensemble weighting is currently the best-performing method on the iMiGUE benchmark.
Reading between the lines
- The ensemble weights are selected on only 777 validation samples, so the 73.213% test figure could be sensitive to that particular validation split; repeated evaluation across different validation splits would show how stable the result is.
- Because the modality extractors are all off-the-shelf, the same weighted late-fusion recipe could transfer to other fine-grained behavior benchmarks, with the relative contribution of each modality possibly shifting.
- Depth and optical flow together add less than one percentage point over skeleton plus RGB plus Taylor, so a cheaper two- or three-stream system with MA-52 pretraining might recover most of the benefit at lower computational cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents MM-Gesture, a multimodal ensemble system for micro-gesture classification on the iMiGUE dataset. The system integrates six modalities (joint, limb, RGB, Taylor-series video, optical flow, depth) using PoseConv3D and Video Swin Transformer backbones, plus MA-52 pretraining for the RGB stream, and combines the resulting probability streams through a weighted late-fusion ensemble. The authors report a top-1 accuracy of 73.213% on the iMiGUE test set, which ranked first in the 3rd MiGA Challenge at IJCAI 2025, and they claim state-of-the-art performance on this benchmark.
Significance. The external competition result is a strong form of validation: the 73.213% figure comes from an official leaderboard, and the paper releases code, which aids reproducibility. The system appears to be a well-engineered combination of existing components, and the empirical gains over prior MiGA challenge entries (roughly +3% over the 2024 winner) are substantial for this benchmark. The main scientific value is the demonstration that a carefully tuned multimodal ensemble, together with MA-52 transfer learning, achieves the current best reported result in the MiGA challenge series. The methodological novelty is modest, however, as the components are off-the-shelf and the fusion is standard late fusion; the claimed state-of-the-art status also requires a broader comparison than the challenge leaderboards alone.
major comments (3)
- [§4.2, Table 1] The claim that MM-Gesture achieves 'state-of-the-art performance' and 'superior performance compared to previous state-of-the-art methods' (abstract and §5) is not supported by the evidence presented. Table 1 compares only the top-3 entries from MiGA challenge leaderboards for 2023–2025. Several published methods that report iMiGUE top-1 accuracy are cited in the related work (e.g., refs. [10], [19], [24]) but are absent from the comparison. If any published method exceeds 73.213% on iMiGUE, the state-of-the-art claim is false, even though the competition ranking itself is an externally verified fact. The authors should add a systematic comparison with all publicly reported iMiGUE results, or explicitly restrict their claim to 'best among MiGA challenge entries.'
- [§3.5, Eq. (7)] The ensemble weights w_i are stated to be 'empirically determined weights obtained via validation-set performance,' but the paper does not report the weight values, the search procedure, or the validation accuracy used to select them. This makes the final 73.213% result, and specifically the 0.569% improvement from the 'optimized multimodal fusion weighting strategy' (Table 2, 72.644% to 73.213%), impossible to reproduce from the paper alone. The authors should report the selected weights, describe the validation-based selection method (e.g., grid search or a learning algorithm), and ideally include an ablation comparing equal weighting against the tuned weights.
- [Table 2] The checkmark layout in Table 2 is ambiguous for the rows with a single backbone name, because the checkmarks are not visually aligned with the column headers in the text, making the modality-to-accuracy mapping unclear (for instance, which single modality yields 65.256%?). Additionally, the final row 'MM-Gesture (Ours)' shows seven checkmarks, while Eq. (7) defines an ensemble of six probability streams; the relationship between the seven table columns (Joint, Limb, RGB, RGB*, Taylor, Flow, Depth) and the six ensemble terms (R+J, R+L, R, T, F, D) needs clarification. The text also reports '72.096%' for the Taylor-inclusive row while Table 2 lists 72.095%; this inconsistency should be fixed.
minor comments (4)
- [§3.1, Eq. (1)] The Taylor-series expansion parameters are not reported: the maximum order K and the temporal window τ used in ℱ_taylor are never specified, although they are free hyperparameters that may affect the Taylor modality's contribution. Please provide the values used in the experiments.
- [§3.1] The selection of the 36 skeleton keypoints from the original 137 is described only as focusing on upper body, hands, and facial joints. Please specify the exact keypoint indices or the selection criterion, since this choice influences the joint and limb modalities.
- [§3.4–§3.5] The symbol R is used for both the RGB input to PoseConv3D (Eq. (5)) and the RGB model in the VideoSwinT stream (Eq. (6)), where the latter is pretrained on MA-52. This notation conflict makes it unclear which RGB probability is used in the ensemble Eq. (7). Please disambiguate (e.g., R_P and R_S or R and R* consistently).
- [§4.1] The paper reports a single evaluation without variance, confidence intervals, or significance tests. The competition leaderboard provides a fixed external result, but for the ablation findings in Table 2, the lack of repeated runs makes it hard to judge whether differences such as 72.227% vs. 72.644% are meaningful. A brief statement acknowledging this limitation would be appropriate.
Circularity Check
No significant circularity: the reported 73.213% test accuracy is an external evaluation on the iMiGUE test split, and the validation-tuned ensemble weights do not make the result derivable from its own inputs.
full rationale
The paper's central claim is an empirical benchmark result, not a derivation. The ensemble weights in Eq. (7) are selected on the 777-sample validation split ('weights set empirically according to validation performance'), but the final top-1 accuracy is reported on the held-out 4,562-sample test set. Therefore the test accuracy is not equivalent to the fitted weights by construction; it remains an independent evaluation. The method combines off-the-shelf architectures (PoseConv3D, VideoSwinT) and externally defined auxiliary modalities (Taylor, optical flow, depth), and the MA-52 pretraining is a use of a public dataset rather than a self-citing proof step. No uniqueness theorem or prior author result is invoked to forbid alternatives or force the ensemble choice. The main limitation is comparative: Table 1 only lists MiGA challenge leaderboard entries, so the broad 'state-of-the-art' phrasing is under-supported relative to all published iMiGUE results. That is a correctness/completeness risk, not circularity. No step in the paper reduces an output to an input by definition, by fitted-parameter renaming, or by a self-citation chain.
Assumptions & free parameters
free parameters (3)
- ensemble_weights =
not reported
- Taylor_expansion_order_and_window =
not reported
- keypoint_subset_selection =
36 of 137 OpenPose keypoints
assumptions (4)
- ad hoc to paper Validation-tuned ensemble weights generalize to the test set
- domain assumption Gaussian heatmap encoding of skeleton keypoints captures micro-gesture structure
- domain assumption Off-the-shelf Taylor, MemFlow, and depth tools produce micro-gesture-relevant features
- domain assumption The official iMiGUE cross-subject split is an unbiased generalization estimate
Cite this review
Pith. "Pith review of MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion." pith.science (2026). https://pith.science/paper/FAZ3VK6Z
@misc{pith2026250708344,
author = {Pith},
title = {Pith review of: MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion},
year = {2026},
howpublished = {\url{https://pith.science/paper/FAZ3VK6Z}},
note = {Machine review of arXiv:2507.08344}
}
read the original abstract
In this paper, we present MM-Gesture, the solution developed by our team HFUT-VUT, which ranked 1st in the micro-gesture classification track of the 3rd MiGA Challenge at IJCAI 2025, achieving superior performance compared to previous state-of-the-art methods. MM-Gesture is a multimodal fusion framework designed specifically for recognizing subtle and short-duration micro-gestures (MGs), integrating complementary cues from joint, limb, RGB video, Taylor-series video, optical-flow video, and depth video modalities. Utilizing PoseConv3D and Video Swin Transformer architectures with a novel modality-weighted ensemble strategy, our method further enhances RGB modality performance through transfer learning pre-trained on the larger MA-52 dataset. Extensive experiments on the iMiGUE benchmark, including ablation studies across different modalities, validate the effectiveness of our proposed approach, achieving a top-1 accuracy of 73.213%. Code is available at: https://github.com/momiji-bit/MM-Gesture.
Figures
Forward citations
Cited by 8 Pith papers
-
iMiGUE-3K: A Large-Scale Benchmark for Micro-Gesture Analysis with Self-Supervised Learning
iMiGUE-3K is the largest in-the-wild micro-gesture video dataset with 3.4K clips and 37M frames from real interviews, supporting self-supervised foundation models and benchmarks that show micro-gestures improve emotio...
-
Spatial-Temporal Decoupled Adapter for Micro-gesture Online Recognition
A decoupled adapter with independent spatial-temporal branches via depthwise convolutions and a dynamic augmentation strategy for long-tail data achieves first place with F1 0.43808 in a micro-gesture recognition challenge.
-
Micro-DualNet: Dual-Path Spatio-Temporal Network for Micro-Action Recognition
Micro-DualNet employs dual ST and TS pathways with entity-level adaptive routing and Mutual Action Consistency loss to achieve competitive results on MA-52 and state-of-the-art on iMiGUE for micro-action recognition.
-
A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition
A multi-modal system combining skeleton/heatmap/RGB features with cross-modal pseudo-labeling and semantic losses achieves 68.13% F1-score and 4th place on the MiGA-IJCAI micro-gesture challenge.
-
Motion Reinforces Appearance: RGB-Skeleton Gated Residual Fusion for Micro-Gesture Online Recognition
DyFADet+ extends a prior detector with gated RGB-skeleton fusion and reports 40.88 F1 on the SMG dataset for micro-gesture online recognition.
-
Self-supervised Learning Matters: A Simple Ensemble Solution for Micro-Gesture Recognition
Ensemble of self-supervised RGB model and supervised models achieves new SOTA of 74.419% on iMiGUE micro-gesture dataset.
-
MAC 2026: Advancing Micro-Action Analysis Towards Fine-Grained Understanding
MAC 2026 reports a three-track micro-action challenge, adding a fine-grained MLLM-based understanding track evaluated on MA-Bench, with top-3 leaderboard results for each track.
-
Rethinking the Role of Feature Engineering and Learning Strategies in Few-Shot Hidden Emotion Recognition
A competition-winning multi-modal model for hidden emotion recognition integrates static and dynamic pose features via cross-attention and MIL pooling while noting representation collapse in vision foundation models o...
Reference graph
Works this paper leans on
-
[10]
K. Li, D. Guo, G. Chen, C. Fan, J. Xu, Z. Wu, H. Fan, M. Wang, Prototypical calibrating ambiguous samples for micro-action recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025, pp. 4815–4823
2025
-
[19]
G. Chen, F. Wang, K. Li, Z. Wu, H. Fan, Y. Yang, M. Wang, D. Guo, Prototype learning for micro-gesture classification, arXiv preprint arXiv:2408.03097 (2024)
arXiv 2024
- [24]
-
[1]
H. Chen, X. Liu, X. Li, H. Shi, G. Zhao, Analyze spontaneous gestures for emotional stress state recognition: A micro-gesture dataset and analysis with deep learning, in: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition, 2019, pp. 1–8
2019
-
[2]
X. Liu, H. Shi, H. Chen, Z. Yu, X. Li, G. Zhao, imigue: An identity-free video dataset for micro- gesture understanding and emotion analysis, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10631–10642
2021
-
[3]
H. Chen, H. Shi, X. Liu, X. Li, G. Zhao, Smg: A micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis, International Journal of Computer Vision 131 (2023) 1346–1366
2023
-
[4]
H. Chen, B. W. Schuller, E. Adeli, G. Zhao, The 2nd challenge on micro-gesture analysis for hidden emotion understanding (miga) 2024: Dataset and results, in: MiGA 2024: Proceedings of IJCAI 2024 Workshop&Challenge on Micro-gesture Analysis for Hidden Emotion Understanding (MiGA
work page 2024
-
[5]
K. Li, D. Guo, M. Wang, Proposal-free video grounding with contextual pyramid network, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2021, pp. 1902–1910
work page 2021
Show all 34 references
-
[6]
F. Wang, K. Li, Y. Nie, Z. Duan, P. Zou, Z. Wu, Y. Wang, Y. Wei, Exploiting ensemble learning for cross-view isolated sign language recognition, arXiv preprint arXiv:2502.02196 (2025)
2025 arXiv
-
[7]
Balazia, P
M. Balazia, P. Müller, Á. L. Tánczos, A. v. Liechtenstein, F. Bremond, Bodily behaviors in social interaction: Novel annotations and state-of-the-art evaluation, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 70–79
2022
-
[8]
D. Li, B. Xing, X. Liu, B. Xia, B. Wen, H. Kälviäinen, Deemo: De-identity multimodal emotion recognition and reasoning, arXiv preprint arXiv:2504.19549 (2025)
2025
-
[9]
Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, H. Hu, Video swin transformer, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 3202–3211
2022
-
[11]
D. Guo, K. Li, B. Hu, Y. Zhang, M. Wang, Benchmarking micro-action recognition: Dataset, methods, and applications, IEEE Transactions on Circuits and Systems for Video Technology 34 (2024) 6238–6252
2024
-
[12]
K. Li, D. Guo, G. Chen, X. Peng, M. Wang, Joint skeletal and semantic embedding loss for micro-gesture classification, arXiv preprint arXiv:2307.10624 (2023)
2023 arXiv
-
[13]
Huang, X
H. Huang, X. Guo, W. Peng, Z. Xia, Micro-gesture classification based on ensemble hypergraph- convolution transformer., in: MiGA@ IJCAI, 2023
2023
-
[14]
K. Li, P. Liu, D. Guo, F. Wang, Z. Wu, H. Fan, M. Wang, Mmad: Multi-label micro-action detection in videos, arXiv preprint arXiv:2407.05311 (2024)
2024 arXiv
-
[15]
K. Li, D. Guo, G. Chen, F. Liu, M. Wang, Data augmentation for human behavior analysis in multi- person conversations, in: Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 9516–9520
2023
-
[16]
J. Gu, K. Li, F. Wang, Y. Wei, Z. Wu, H. Fan, M. Wang, Motion matters: Motion-guided modu- lation network for skeleton-based micro-action recognition, in: Proceedings of the 33rd ACM International Conference on Multimedia, 2025
2025
-
[17]
S. Sun, D. Liu, J. Dong, X. Qu, J. Gao, X. Yang, X. Wang, M. Wang, Unified multi-modal unsupervised representation learning for skeleton-based action understanding, in: Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 2973–2984
2023
-
[18]
J. Dong, S. Sun, Z. Liu, S. Chen, B. Liu, X. Wang, Hierarchical contrast for unsupervised skeleton- based action representation learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023, pp. 525–533
2023
-
[20]
Huang, Y
H. Huang, Y. Wang, L. Kerui, Z. Xia, Multi-modal micro-gesture classification via multiscale heterogeneous ensemble network, MiGA@ IJCAI (2024)
2024
-
[21]
H. Duan, Y. Zhao, K. Chen, D. Lin, B. Dai, Revisiting skeleton-based action recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2969–2978
2022
-
[22]
F. Wang, D. Guo, K. Li, M. Wang, Eulermormer: Robust eulerian motion magnification via dynamic filtering within transformer, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 5345–5353
2024
-
[23]
F. Wang, D. Guo, K. Li, Z. Zhong, M. Wang, Frequency decoupling for motion magnification via multi-level isomorphic architecture, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18984–18994
2024
-
[25]
L. Wang, X. Yuan, T. Gedeon, L. Zheng, Taylor videos for action recognition, in: Forty-first International Conference on Machine Learning, 2024
2024
-
[26]
Q. Dong, Y. Fu, Memflow: Optical flow estimation and prediction with memory, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19068–19078
2024
-
[27]
S. Chen, H. Guo, S. Zhu, F. Zhang, Z. Huang, J. Feng, B. Kang, Video depth anything: Consistent depth estimation for super-long videos, arXiv preprint arXiv:2501.12375 (2025)
2025 arXiv
-
[28]
H. Xu, L. Cheng, Y. Wang, S. Tang, Z. Zhong, Towards fine-grained emotion understanding via skeleton-based micro-gesture recognition, arXiv preprint arXiv:2506.12848 (2025)
2025 arXiv
-
[29]
Y. Xu, L. Zhu, Y. Yang, Mc-bench: A benchmark for multi-context visual grounding in the era of mllms, arXiv preprint arXiv:2410.12332 (2024)
2024 arXiv
-
[30]
Y. Xu, L. Zhu, Y. Yang, Gg-editor: Locally editing 3d avatars with multimodal large language model guidance, in: Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 10910–10919
2024
-
[31]
F. Liu, K. Li, Z. Zhong, W. Jia, B. Hu, X. Yang, M. Wang, D. Guo, Depth matters: Spatial proximity- based gaze cone generation for gaze following in wild, ACM Transactions on Multimedia Comput- ing, Communications and Applications 20 (2024) 1–24
2024
-
[32]
J. Zhao, F. Wang, K. Li, Y. Wei, S. Tang, S. Zhao, X. Sun, Temporal-frequency state space duality: An efficient paradigm for speech emotion recognition, in: ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing, 2025, pp. 1–5
2025
-
[33]
W. Qian, K. Li, D. Guo, B. Hu, M. Wang, Cluster-phys: Facial clues clustering towards efficient remote physiological measurement, in: Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 330–339
2024
-
[2024]
co-located with 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024), 2024
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.