REVIEW 4 major objections 4 minor 39 references
VirtualFencer: Generating Fencing Bouts based on Strategies Extracted from In-the-Wild Videos
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read VirtualFencer recovers 3D fencing motion and right-of-way strategy from unlabeled broadcast video, and professional saber fencers rate its self-play touches as indistinguishable from ground-truth touches.
desk verdict A solid systems paper with real expert evaluation, but the core strategy-extraction claim rests on an unvalidated displacement-based priority heuristic. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a three-part pipeline: (1) a homography-corrected pose-estimation stage that maps both fencers' global positions onto the canonical 14-meter piste, using WHAM for SMPL pose and SAM2-prompted tracking of piste lines; (2) a two-stage $k$-means clustering over 20-frame subclips embedded with TMR features, dominant-arm axis-angle joints, and distance statistics, yielding 30 labeled skill clusters; and (3) a priority-aware strategy model: three separate first-order transition matrices conditioned on the priority state (M-M, P-NP, NP-P), combined with a Gaussian distance weight at inference time, and a simulation state machine that updates priority by scoring lights, finishing actions, or displacement fallback, and terminates on out-of-bounds, crash, touch, or terminal action.
What would settle it
Take a sample of touches from the 40 collected bouts, compare the heuristic priority labels from Section 5.3 against the official referee's right-of-way decisions (recoverable from the scoring lights and the bout outcome), and measure the disagreement rate; a high disagreement rate around exchanges would show the strategy model is conditioned on noisy labels. A stronger test would re-run the self-play user study with referee-verified priority labels instead of the heuristic and check whether the indistinguishability result persists.
Extended reading notes
Core claim
The authors claim that a fully unsupervised pipeline from monocular broadcast video to a strategy-conditioned generative model of fencing touches works: they extract about 1.5 hours of motion from 40 senior-level bouts, correct the global translations of the off-the-shelf pose estimator using a homography aligned to the fencing piste, cluster motion embeddings into 30 discrete skill clusters, and fit three priority-mode transition distributions of the form $P(u_t | u_{t-1}, v_{t-1}, d_t)$ with a Gaussian distance weighting. The learned model is then used in a state-machine simulation that generates complete touches. The user study shows professional fencers split their preferences evenly between self-play touches and ground-truth touches, which the authors take as validation that the extracted strategy—including right-of-way dynamics—is correct.
Load-bearing premise
The load-bearing premise is that right-of-way can be inferred from which fencer's global displacement is larger by a fixed threshold of 0.3 meters, with no validation against actual referee calls; if this heuristic mislabels priority states, the transition tables and the entire strategy model learn from incorrect context.
Editorial extensions
If this is right
- If the strategy model is correct, the same skill-cluster vocabulary and transition model can generate novel touches for any pairing of fencers, as demonstrated by self-play, play against a recorded real fencer, and interactive play against a professional.
- The system provides a working interactive opponent for training purposes, with experts rating its understanding of priority, action diversity, and distance handling at 4, 4.25, and 3.75 out of 5 respectively.
- The next-action prediction experiment shows the model's single-step choices are ranked between ground truth and random, while the full self-play chains are rated as good as ground truth, suggesting that the tactical quality emerges from the sequential composition rather than from any single action.
- The termination conditions and priority-update rules allow the simulation to produce complete touches that respect the basic structure of real saber fencing, including scoring lights and right-of-way forfeiture on misses.
Reading between the lines
- The single-step Markov assumption—conditioning only on the previous actions of both fencers—is likely the main bottleneck; the experts' comments about the system lacking the 'why' behind reactions suggest that extending the model to longer contexts or opponent-modeling would be a natural and testable improvement.
- The priority heuristic (displacement difference with a fixed threshold of $\delta = 0.3$) is the most fragile link in the chain; replacing it with referee-verified right-of-way labels, which are knowable from official scoring, would directly test how much of the learned strategy depends on this assumption.
- The success of the self-play evaluation suggests that broadcast video is a sufficient data source for tactical sports modeling; a similar pipeline could be applied to other rule-structured dueling sports with global positioning cues, though that remains an open extension rather than a claim of this paper.
- The fixed 20-frame subclip windows may mask compound actions such as reprise or remise; an adaptive segmentation that lets one fencer take multiple actions while the opponent takes one could recover additional tactical richness and address the experts' noted limitation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. VirtualFencer proposes a pipeline that reconstructs 3D fencing motion from in-the-wild broadcast videos, discovers low-level action clusters in an unsupervised manner, and models high-level fencing strategy as priority-conditioned transition probabilities over those clusters. The learned model is then used to generate new fencing touches via a state-machine simulation, and the paper evaluates the results through three user studies: next-action judgment, self-play bout ranking, and interactive play against professional fencers. The authors report that expert fencers cannot distinguish self-play touches from ground-truth touches, which they interpret as validation of the extracted strategy.
Significance. If the central claim holds, the paper would be a notable demonstration that a data-driven system can recover both low-level motion skills and high-level tactical structure from unlabeled video, without manual action annotation or motion capture. Strengths of the manuscript include a carefully designed domain-aware pipeline, a substantial video collection effort, and a user-study protocol with expert fencers that directly targets the realism of the generated behavior. The paper also is honest about several limitations, including cluster confusion due to body size and motion discontinuities in generated play. The main gaps are in the validation of the priority-annotation heuristic and in the statistical support for the user-study conclusions; these issues prevent the current version from fully establishing the strategy-extraction claim.
major comments (4)
- [Sec. 5.3, Sec. 6.3] The priority labels that condition the transition matrices in Eq. (1) are computed solely from the displacement threshold δ=0.3, with no agreement check against referee calls or expert ratings. In sabre, right-of-way can be held while retreating or lost after a miss, so a purely positional heuristic is likely to mislabel many 20-frame windows; each empirical distribution P_raw(u_t | u_{t-1}, v_{t-1}) is therefore estimated under an incorrect conditioning context. This is load-bearing because the claim of recovering true strategy depends on these labels. Please provide a validation of the priority labels (e.g., agreement statistics against a human expert or a referee) or an ablation with an alternative priority estimator.
- [Sec. 7.1, Table 2] Clustering accuracy is estimated from only 10 manually checked samples per cluster, and several clusters are barely above chance (C.10 at 0.4, C.21 at 0.5). The low-level skills are the vocabulary over which the strategy model is defined, so noisy clusters propagate into the transition statistics. Please report per-cluster accuracy with larger sample sizes and explain how poorly performing clusters are handled in the strategy model; this would materially strengthen the claim that the system recovers meaningful fencing actions.
- [Sec. 7.3, Table 1] The user study is presented as validation that experts 'split their preferences evenly' between VirtualFencer and ground truth, but the paper reports only raw vote counts and no significance test. A chi-square goodness-of-fit or an appropriate paired-comparison test is needed to support the claim. Without it, the observed counts (e.g., 218 first-place votes for VirtualFencer vs. 177 for ground truth in the second study) cannot be interpreted as 'even'; this is load-bearing because the paper's main validation message rests on this comparison.
- [Sec. 6.4.1] The simulation's priority update uses scoring-light metadata and manually selected finishing-action clusters in addition to the displacement rule, so the end-to-end user study does not isolate the Section 5.3 annotation. Consequently, the positive results in Sections 7.3.2 and 7.3.3 cannot validate the displacement-based priority heuristic, and the paper should either test the components separately or temper the claim that the strategy extraction is validated end-to-end.
minor comments (4)
- [Eq. (1)] Equation (1) uses a proportionality sign but does not explicitly define the normalization over actions; please state that sampling normalizes the product.
- [Sec. 5.1] The paper reports about 1.5 hours of fencing actions but does not state the number of touches or the number of 20-frame samples; including these statistics would make the dataset scale and the empirical counts more transparent.
- [Table 2] The term 'Off the line' in the action labels is not standard fencing terminology in this context; please clarify what this category denotes.
- [Sec. 7.3.3] The third user study involves only four participants; while this is a common exploratory setup, the qualitative conclusions should be presented as anecdotal rather than as statistically representative.
Circularity Check
Self-play 'validation' resamples the empirical transition distribution fitted to the same corpus; headline strategy-extraction evidence is in-sample, though the interactive expert study adds independent support.
-
fitted input called prediction
[Section 6.3 (Eq. 1), Section 6.4, Section 7.3.2, Abstract]
"We use the fencing trajectories in our dataset to estimate the raw transition distribution. Given sequences of the form {(u0,v0,d0),(u1,v1,d1),...,(uT,vT,dT)}, we compute the raw transition distribution P_raw(u_t | u_{t-1}, v_{t-1}) from empirical frequency counts... users split their preferences evenly between touches produced by VirtualFencer’s self-play and ground truth touches, while clearly preferring both methods to the random baseline."
The 'ground truth' touches and the next-action contexts used in the user studies come from the same 40-bout corpus (Sec. 5.1) from which P_raw is estimated by empirical frequency counts (Sec. 6.3). The simulator (Sec. 6.4) samples actions directly from that P_raw and retrieves clips from the same clusters. Consequently, a self-play touch is a Monte Carlo draw from the very distribution fitted to the training touches, so its first-order transition statistics match ground truth by construction. The headline result that experts cannot distinguish the two (Sec. 7.3.2) and the abstract's conclusion that the strategy extraction is 'validated' are therefore in-sample resampling checks, not independent predictions.
full rationale
The central derivation is a standard empirical transition model: P_raw is estimated by counting transitions in the collected videos, and simulation samples from it. That is not circular by itself. The circularity enters when the paper presents the self-play user study as validating the extracted strategy: because generated touches are sampled from P_raw and the 'ground truth' touches are from the same corpus used to estimate P_raw, the expert preference is an in-sample resampling fidelity check, statistically forced at the level of transition frequencies. The first 'next action prediction' study is similarly in-sample: the model is queried on contexts from the same dataset that supplied the empirical counts. The interactive professional-fencer study is a genuinely external interaction and gives the central claim some independent content, so this is partial rather than total circularity. The displacement-based priority heuristic (Sec. 5.3) is an additional unvalidated assumption that conditions both training and simulation, but it is a correctness risk rather than a by-construction reduction. No self-citation chain or imported uniqueness theorem is load-bearing.
Assumptions & free parameters
free parameters (6)
- delta (priority displacement threshold) =
0.3
- sigma (distance weighting sensitivity) =
0.5
- tau (crash distance threshold) =
1.5 meters
- scoring distance threshold =
2 meters
- action window length =
20 frames
- number of clusters =
30
assumptions (5)
- domain assumption The panning camera has fixed position and zoom, and the piste lines can be tracked to compute a homography that projects fencers into global coordinates.
- domain assumption Fencing actions are approximately 20 frames long, so 20-frame subclips capture single actions.
- ad hoc to paper A single step of history is sufficient to model fencing transitions (Markov assumption).
- domain assumption The manually labeled cluster labels (e.g., advance, retreat, lunge) are reliable ground truth for evaluating clustering accuracy.
- ad hoc to paper The displacement-based priority heuristic approximates real right-of-way.
Cite this review
Pith. "Pith review of VirtualFencer: Generating Fencing Bouts based on Strategies Extracted from In-the-Wild Videos." pith.science (2026). https://pith.science/paper/5VRU2M6U
@misc{pith2026250700261,
author = {Pith},
title = {Pith review of: VirtualFencer: Generating Fencing Bouts based on Strategies Extracted from In-the-Wild Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/5VRU2M6U}},
note = {Machine review of arXiv:2507.00261}
}
read the original abstract
Fencing is a sport where athletes engage in diverse yet strategically logical motions. While most motions fall into a few high-level actions (e.g. step, lunge, parry), the execution can vary widely-fast vs. slow, large vs. small, offensive vs. defensive. Moreover, a fencer's actions are informed by a strategy that often comes in response to the opponent's behavior. This combination of motion diversity with underlying two-player strategy motivates the application of data-driven modeling to fencing. We present VirtualFencer, a system capable of extracting 3D fencing motion and strategy from in-the-wild video without supervision, and then using that extracted knowledge to generate realistic fencing behavior. We demonstrate the versatile capabilities of our system by having it (i) fence against itself (self-play), (ii) fence against a real fencer's motion from online video, and (iii) fence interactively against a professional fencer.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Tadeusz Bober, Alicja Rutkowska-Kucharska, Sebastian Jaroszczuk, Maciej Barabasz, and Wojciech Woznica. 2016. Original research papers. Kinematic characterisation of the lunge and the fleche in epee fencing: two case studies. Polish Journal of Sport and Tourism 23, 4 (2016), 181
work page 2016
-
[2]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi- person 2D Pose Estimation Using Part Affinity Fields. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 1302–1310. doi:10.1109/CVPR. 2017.143
doi:10.1109/cvpr 2017
-
[3]
Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. 2021. Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video. In Conference on Computer Vision and Pattern Recognition (CVPR)
work page 2021
-
[4]
Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran. 2018. Detect-and-track: Efficient pose estimation in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition . 350–359
work page 2018
-
[5]
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa*, and Jitendra Malik*. 2023. Humans in 4D: Reconstructing and Tracking Humans with Transformers. In International Conference on Computer Vision (ICCV)
work page 2023
-
[6]
Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. 2018. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition . 7297–7306
work page 2018
-
[7]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision . 2961–2969
2017
-
[8]
Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres, and Bernt Schiele. 2017. Arttrack: Articulated multi- person tracking in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6457–6465
work page 2017
Show all 39 references
-
[9]
Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele. 2016. DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model. Springer International Publishing, 34–50. doi:10.1007/978-3- 319-46466-4_3
2016 doi
-
[10]
Umar Iqbal, Anton Milan, and Juergen Gall. 2017. Posetrack: Joint multi-person pose estimation and tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2011–2020
2017
-
[11]
Black, David W
Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. 2018. End-to-end Recovery of Human Shape and Pose. In Computer Vision and Pattern Recognition (CVPR)
2018
-
[12]
Zhang, Panna Felsen, and Jitendra Malik
Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik. 2019. Learning 3D Human Dynamics from Video. In Computer Vision and Pattern Recognition (CVPR)
2019
-
[13]
Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. 2020. VIBE: Video Inference for Human Body Pose and Shape Estimation. In The IEEE Conference Zhiyin Lin, Purvi Goel, Joy Yun, C. Karen Liu, and Joao Pedro Araujo on Computer Vision and Pattern Recognition (CVPR)
2020
-
[14]
Muhammed Kocabas, Ye Yuan, Pavlo Molchanov, Yunrong Guo, Michael Black, Otmar Hilliges, Jan Kautz, and Umar Iqbal. 2024. PACE: Human and Camera Motion Estimation from in-the-wild Videos. In International Conference on 3D Vision (3DV). arXiv:2310.13768
2024 arXiv
-
[15]
Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion graphs. ACM Trans. Graph. 21, 3 (July 2002), 473–482. doi:10.1145/566654.566605
2002
-
[16]
Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard. 2002. Interactive control of avatars animated with human motion data. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 491–500
2002
-
[17]
Jiefeng Li, Siyuan Bian, Qi Liu, Jiasheng Tang, Fan Wang, and Cewu Lu. 2023. NIKI: Neural Inverse Kinematics with Invertible Neural Networks for 3D Hu- man Pose and Shape Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
2023
-
[18]
Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, and Cewu Lu. 2023. HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery. arXiv preprint arXiv:2304.05690 (2023)
2023 arXiv
-
[19]
Alireza Golestaneh, and Kris M
Zhengyi Luo, S. Alireza Golestaneh, and Kris M. Kitani. 2020. 3D Human Motion Estimation via Motion Compression and Refinement. In Proceedings of the Asian Conference on Computer Vision (ACCV)
2020
-
[20]
Troje, Gerard Pons-Moll, and Michael J
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. 2019. AMASS: Archive of Motion Capture as Surface Shapes. In International Conference on Computer Vision . 5442–5451
2019
-
[21]
Filip Malawski. 2018. Real-time first person perspective tracking and feedback system for weapon practice support in fencing. In Applications of Intelligent Systems. IOS Press, 79–88
2018
-
[22]
Filip Malawski. 2022. Immersive feedback in fencing weapon practice using mixed reality. Computer Science 23, 1 (March 2022). doi:10.7494/csci.2022.23.1.4570
2022 doi
-
[23]
Filip Malawski and Bogdan Kwolek. 2018. Recognition of action dynamics in fencing using multimodal cues. Image and Vision Computing 75 (2018), 1–10. doi:10.1016/j.imavis.2018.04.005
2018 doi
-
[24]
Filip Malawski and Bogdan Kwolek. 2019. Improving multimodal action repre- sentation with joint motion history context. Journal of Visual Communication and Image Representation 61 (2019), 198–208. doi:10.1016/j.jvcir.2019.03.026
2019 doi
-
[25]
Kevin C Moore, Frances ME Chow, and John YH Chow. 2015. Novel lunge biomechanics in modern Sabre fencing. Procedia engineering 112 (2015), 473– 478
2015
-
[26]
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. 2019. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVP...
2019
-
[27]
Black, and Gül Varol
Mathis Petrovich, Michael J. Black, and Gül Varol. 2023. TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis. In International Con- ference on Computer Vision (ICCV)
2023
-
[28]
Henning, and Stefan Leutenegger
Simon Schaefer, Dorian F. Henning, and Stefan Leutenegger. 2023. GloPro: Globally-Consistent Uncertainty-Aware 3D Human Pose Estimation & Tracking in the Wild. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 3803–3810. doi:10.1109/IROS55552....
2023
-
[29]
Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J. Black. 2024. WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion. InIEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)
2024
-
[30]
Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand Key- point Detection in Single Images Using Multiview Bootstrapping . In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE Computer Society, Los Alamitos, CA, USA, 4645–4653. doi:...
2017 doi
-
[31]
Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges. 2017. Thin-slicing network: A deep structured model for pose estimation in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4220–4229
2017
-
[32]
Jonathan Taylor, Jamie Shotton, Toby Sharp, and Andrew Fitzgibbon. 2012. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In 2012 IEEE conference on computer vision and pattern recognition . IEEE, 103–110
2012
-
[33]
Wen-Li Wei, Jen-Chun Lin, Tyng-Luh Liu, and Hong-Yuan Mark Liao. 2022. Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2022
-
[34]
Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. 2023. De- coupling Human and Camera Motion from Videos in the Wild. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2023
-
[35]
Haotian Zhang, Cristobal Sciutto, Maneesh Agrawala, and Kayvon Fatahalian
-
[36]
Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. 2023. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
2023
-
[37]
Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. [n. d.]. Learning Physically Simulated Tennis Skills from Broadcast Videos. ACM Trans. Graph. ([n. d.]), 14 pages. doi:10.1145/3592408
-
[38]
It understands right of way, and it knows when to attack and defend. This logic takes most beginning fencers many months to properly learn
Kevin Zhu, Alexander Wong, and John McPhee. 2022. FenceNet: Fine-Grained Footwork Recognition in Fencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops . 3589–3598. VirtualFencer: Generating Fencing Bouts based on Strateg...
2022
-
[2021]
ACM Trans
Vid2Player: Controllable Video Sprites That Behave and Appear Like Professional Tennis Players. ACM Trans. Graph. 40, 3, Article 24 (May 2021), 16 pages. doi:10.1145/3448978
2021 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.