REVIEW 4 major objections 4 minor 24 references
Proximal Control of UAVs with Federated Learning for Human-Robot Collaborative Domains
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper aims to show that a federated-learning-trained LSTM classifier can give a UAV proximal gesture control, and that a second UAV prevents occlusion by taking over detection when the first loses sight of the operator.
desk verdict The abstract's >96% real-robot accuracy claim is contradicted by the paper's own 68.67% real-time result; the real contribution is a working FL-LSTM multi-UAV handover demo and dataset, but the paper overstates it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-layer LSTM followed by three dense layers (with dropout) that operates on a sliding window of 60 frames of 33 MediaPipe body keypoints (x, y, z coordinates), and is trained across clients by federated averaging. The occlusion countermeasure is a priority check: the secondary drone's commands are ignored whenever the primary drone detects the operator, so the second viewpoint acts as a fallback transmitter. This combination—the streaming keypoint window and the priority-checked viewpoint handover—carries the argument.
What would settle it
A controlled reproduction that runs the same trained model on the same drone platforms while varying operator distance, lighting, and camera quality in the ranges described in Section VI-B, and measures end-to-end command accuracy, would settle whether the offline accuracy transfers; if accuracy stays near 0.69 rather than above 0.96, the paper's headline accuracy claim is not supported.
Extended reading notes
Core claim
The central claim is that an LSTM with two recurrent layers and three dense layers, trained with federated averaging across drone-mounted clients, classifies 13 gesture/action classes from MediaPipe body keypoints well enough to command a UAV, and that a second drone can take over command when the primary drone loses sight of the operator. The paper reports training and validation accuracy of 96–97% on its own dataset (Table III) and a real-time system accuracy of 0.6867, with precision 0.93 and F1 0.83 in the real-robot trials. The authors state that this is the first work applying federated learning to proximal control of UAVs in human-robot interaction.
Load-bearing premise
The load-bearing premise is that the high offline validation accuracy of 96–97% transfers to real-robot closed-loop operation, despite the paper reporting a real-time accuracy of 0.6867 in Section VI-B.
Editorial extensions
If this is right
- Gesture-based proximal control of UAVs can be trained without centralizing operator video data, since only model weights are shared across drones.
- Multi-robot teams can maintain continuous command of a drone by viewpoint handover, because the secondary robot's detection covers the primary robot's blind spots.
- The same federated architecture extends to more than two drones or cameras, as the authors note, so larger swarms could share a single action model.
- A streaming action classifier with a 10.4-second command window can be deployed onboard with modest compute, as demonstrated on the X500 drone.
- The work offers a comparison between a literature dataset and a new three-view dataset, providing a baseline for future gesture-control UAV research.
Reading between the lines
- If federated training converges in only three rounds with two clients, the value of FL here may be less about privacy or communication savings than about pooling training signal from multiple viewpoints; a direct comparison against a centrally trained model on the same data would separate these effects.
- The gap between 96–97% offline validation accuracy and 0.6867 real-time accuracy suggests that robustness to lighting, distance, and sensor noise is the actual bottleneck; a controlled test that varies these conditions systematically could guide targeted data augmentation.
- The 60-frame sliding window and the 10-second wait between commands restrict the system to discrete, slow command sequences; extending to continuous velocity commands would require a different action set and temporal decoding, a natural next step the paper does not address.
- The novelty claim of first use of federated learning in proximal UAV control rests on the completeness of the cited literature; none of the gesture-control works referenced use FL, which supports the claim but does not prove it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a federated-learning (FedAvg) pipeline for LSTM-based action recognition from Mediapipe keypoints, used for proximal gesture control of a UAV, with a second UAV maintaining the human operator in view to avoid occlusions. The authors create a second gesture dataset, validate their model on two datasets, and report a real-robot demonstration with three clients and two drones. The abstract claims that experiments with real robots achieved accuracy greater than 96%, and the paper claims to be the first work using federated learning in HRI for proximal control of UAVs.
Significance. If the claims were supported, the paper would offer a practical demonstration of onboard federated LSTM action recognition for multi-UAV proximal control, with a public dataset and a real-robot occlusion-handover experiment. These are genuine strengths: the authors deployed two different UAV platforms, used standard FL tooling (Flower), report a confusion matrix, and provide a video link. However, because the headline real-robot accuracy is directly contradicted by the reported 68.67% real-time accuracy and because the FL component is not compared with any non-FL or centralized baseline, the significance of the contribution as stated is not established.
major comments (4)
- [Abstract and Section VI-B] The abstract's claim that 'experiments with real robots achieving an accuracy greater than 96%' is directly contradicted by Section VI-B, which reports that 'in the real-time experiments, the Accuracy of our system was 0.6867.' Table III's 0.96/0.97 figures are model validation accuracies on dataset2, not closed-loop real-robot results. Since the paper's own real-robot experiment is the only direct evidence for the headline claim, the abstract-level result is unsupported and must be corrected.
- [Sections V and VI-A] The federated-learning contribution is not evaluated against any relevant baseline. Section VI-A states that each client used the same dataset, rather than the non-IID partitions that motivate FL in Section V, and the paper reports no comparison with centralized training or with local-only training. Consequently, the statements that FL enables distributed training and accelerates convergence are not supported by the experiments.
- [Section VI-B] The 'absence of overfitting' argument is not a valid test: distinguishing a wrong action prediction from a missed detection addresses error modes, not the generalization gap between training and deployment. The paper does not report a held-out real-robot protocol or per-trial variability, so the drop from 96-97% validation accuracy to 68.67% real-time accuracy remains unexplained and is attributed only to environmental factors.
- [Section IV-2 and Table I] The created dataset is described as having 33 unique actions and 6 unique static gestures after multiplying 11 actions and 2 gestures by three viewpoints, yet the architecture in Table I has a 13-unit softmax output and Section III-B describes commands for 13 classes. If viewpoint-specific labels are used, the output layer cannot represent them; if they are mapped back to 13 classes, the '33 unique actions' statement is misleading. The label mapping needs clarification.
minor comments (4)
- [Fig. 12] The captions alternate between 'Hover' and 'Have command' for what appears to be the same action; please standardize the action names.
- [References] Reference [19] is cited for OpenPose but the arXiv identifier 2007.14390 actually belongs to the Flower framework, which is also cited as [24]; the OpenPose citation should be corrected.
- [Section III-B] The phrase 'In resume' should be 'In summary'.
- [Equation (1)] The symbol R is used both for the rotation matrix and for the set of unit vectors in equation (3); please use distinct notation to avoid ambiguity.
Circularity Check
No circular derivation: the paper is an empirical ML/robotics study, and its abstract accuracy claim, while internally inconsistent with the reported real-time accuracy, is not a definitional reduction.
full rationale
The paper contains no derivation chain whose output is equivalent to its input by construction. The proposed system is an LSTM action classifier trained with federated averaging; performance is measured empirically on two datasets (Perera et al. and the authors' own dataset) and in real-robot experiments. The dataset augmentation equation (1) defines a rotation operation on frames; it does not encode the classification outcome. Tables II and III report training/validation loss and accuracy; these are standard empirical metrics, not fitted parameters renamed as predictions. The real-robot accuracy of 0.6867 is reported separately in Section VI-B, and the abstract's '>96%' figure appears to be drawn from the dataset-2 validation accuracy of Table III rather than from the closed-loop real-time result. This is a serious reporting inconsistency and an evidentiary weakness, but it is not circularity because the validation metric is not defined in terms of the real-robot claim. The FL implementation follows the standard FedAvg algorithm from McMahan et al. [7], and the MRS system [23] is used as a component; these self-citations are not load-bearing in the sense of justifying a prediction by the authors' own prior unverified result. The 'first work using FL in HRI for proximal control' claim is a novelty statement, not a mathematical derivation. No step reduces to its own input, so the circularity score is 0.
Assumptions & free parameters
free parameters (7)
- FedAvg rounds =
5
- Training epochs / early stopping patience =
750 / 300
- Action recognition window =
60 frames
- Moving average window =
3 actions
- Command wait/debounce period =
10 seconds
- Data augmentation rotation range =
-15 to +15 degrees
- Network architecture sizes =
LSTM 30/64, dense 64/32, output 13
assumptions (4)
- domain assumption FedAvg with only five rounds and two or three clients converges to a useable global model
- domain assumption The DJI UAV hovering at a fixed position keeps the operator in its field of view whenever the X500 loses sight
- domain assumption Mediapipe skeleton keypoints from both UAV viewpoints are sufficiently consistent for a single shared LSTM classifier
- ad hoc to paper Validation accuracy on the created dataset transfers to real-robot operation
Cite this review
Pith. "Pith review of Proximal Control of UAVs with Federated Learning for Human-Robot Collaborative Domains." pith.science (2026). https://pith.science/paper/POT5VFEA
@misc{pith2026241202863,
author = {Pith},
title = {Pith review of: Proximal Control of UAVs with Federated Learning for Human-Robot Collaborative Domains},
year = {2026},
howpublished = {\url{https://pith.science/paper/POT5VFEA}},
note = {Machine review of arXiv:2412.02863}
}
read the original abstract
The human-robot interaction (HRI) is a growing area of research. In HRI, complex command (action) classification is still an open problem that usually prevents the real applicability of such a technique. The literature presents some works that use neural networks to detect these actions. However, occlusion is still a major issue in HRI, especially when using uncrewed aerial vehicles (UAVs), since, during the robot's movement, the human operator is often out of the robot's field of view. Furthermore, in multi-robot scenarios, distributed training is also an open problem. In this sense, this work proposes an action recognition and control approach based on Long Short-Term Memory (LSTM) Deep Neural Networks with two layers in association with three densely connected layers and Federated Learning (FL) embedded in multiple drones. The FL enabled our approach to be trained in a distributed fashion, i.e., access to data without the need for cloud or other repositories, which facilitates the multi-robot system's learning. Furthermore, our multi-robot approach results also prevented occlusion situations, with experiments with real robots achieving an accuracy greater than 96%.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Robots in the loop: Telepresence robots in everyday life,
K. M. Tsui, S. V on Rump, H. Ishiguro, L. Takayama, and P. Vicars, “Robots in the loop: Telepresence robots in everyday life,” inACM/IEEE HRI, Mar. 2012, pp. 317–318, iSSN: 2167-2148
work page 2012
-
[2]
Human Gesture Robot Control Using a Camera/Accelerometer-in-Palm Sensor,
J. Floersch and P. Y . Li, “Human Gesture Robot Control Using a Camera/Accelerometer-in-Palm Sensor,”IFAC-PapersOnLine, vol. 54, no. 20, pp. 284–289, Jan. 2021
work page 2021
-
[3]
Deceptive actions to improve the attribution of rationality to playing robotic agents,
E. de Oliveira, L. Donadoni, S. Boriero, and A. Bonarini, “Deceptive actions to improve the attribution of rationality to playing robotic agents,”International Journal of Social Robotics, vol. 13, pp. 391–405, 2021
work page 2021
-
[4]
UA V Vision-Based Nonlinear Formation Control Applied to Inspection of Electrical Power Lines,
T. Uzakov, T. P. Nascimento, and M. Saska, “UA V Vision-Based Nonlinear Formation Control Applied to Inspection of Electrical Power Lines,” in2020 ICUAS). Athens, Greece: IEEE, Sep. 2020, pp. 1301– 1308
work page 2020
-
[5]
A. Chaudhary, T. Nascimento, and M. Saska, “Controlling a swarm of unmanned aerial vehicles using full-body k-nearest neighbor based action classifier.” IEEE, 6 2022, pp. 544–551
work page 2022
-
[6]
Federated learning in robotic and autonomous systems,
Y . Xianjia, J. P. Queralta, J. Heikkonen, and T. Westerlund, “Federated learning in robotic and autonomous systems,”Procedia Computer Sci- ence, vol. 191, pp. 135–142, 2021
work page 2021
-
[7]
Communication-efficient learning of deep networks from decentralized data,
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics. PMLR, 2017, pp. 1273– 1282
2017
-
[8]
Decentralized Federated Learning for UAV Networks: Architecture, Challenges, and Opportunities
Y . Qu, H. Dai, Y . Zhuang, J. Chen, C. Dong, F. Wu, and S. Guo, “Decentralized Federated Learning for UA V Networks: Architecture, Challenges, and Opportunities,” Aug. 2021, arXiv.2104.07557 [cs, stat]
work page Pith review arXiv 2021
Show all 24 references
-
[9]
Gesture recognition using skeleton data with weighted dynamic time warping,
S. Celebi, A. S. Aydin, T. T. Temiz, and T. Arici, “Gesture recognition using skeleton data with weighted dynamic time warping,” vol. 1, 2013, pp. 620–625
2013
-
[10]
A differential evolution approach to optimize weights of dynamic time warping for multi-sensor based gesture recognition,
J. Rwigema, H. R. Choi, and T. Kim, “A differential evolution approach to optimize weights of dynamic time warping for multi-sensor based gesture recognition,”Sensors (Switzerland), vol. 19, 3 2019
2019
-
[11]
Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model,
N. L. Hakim, T. K. Shih, S. P. Kasthuri Arachchi, W. Aditya, Y .-C. Chen, and C.-Y . Lin, “Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model,”Sensors, vol. 19, no. 24, p. 5429, Jan. 2019
2019
-
[12]
Human activity recognition using federated learning,
K. Sozinov, V . Vlassov, and S. Girdzijauskas, “Human activity recognition using federated learning,” in2018 IEEE ISPA/IUCC/BDCloud/SocialCom/SustainCom, 2018, pp. 1103–1111
2018
-
[13]
Drone.io: A Gestural and Visual Interface for Human- Drone Interaction,
J. R. Cauchard, A. Tamkin, C. Y . Wang, L. Vink, M. Park, T. Fang, and J. A. Landay, “Drone.io: A Gestural and Visual Interface for Human- Drone Interaction,” inACM/IEEE HRI. Daegu, Korea (South): IEEE, Mar. 2019, pp. 153–162
2019
-
[14]
Real-Time Human- UA V Interaction: New Dataset and Two Novel Gesture-Based Interacting Systems,
M. A. Kassab, M. Ahmed, A. Maher, and B. Zhang, “Real-Time Human- UA V Interaction: New Dataset and Two Novel Gesture-Based Interacting Systems,”IEEE Access, vol. 8, pp. 195 030–195 045, 2020
2020
-
[15]
Real-Time Human Detection and Gesture Recognition for On-Board UA V Rescue,
C. Liu and T. Szir ´anyi, “Real-Time Human Detection and Gesture Recognition for On-Board UA V Rescue,”Sensors, vol. 21, no. 6, p. 2180, Jan. 2021
2021
-
[16]
Microsoft coco: Common objects in con- text,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in con- text,” inComputer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 201...
2014
-
[17]
Uav-gesture: A dataset for uav control and gesture recognition,
A. G. Perera, Y . W. Law, and J. Chahl, “Uav-gesture: A dataset for uav control and gesture recognition,” inComputer Vision – ECCV 2018 Workshops, L. Leal-Taix ´e and S. Roth, Eds. Springer International Publishing, 2019, pp. 117–128
2018
-
[18]
Drone-Action: An Outdoor Recorded Drone Video Dataset for Action Recognition,
——, “Drone-Action: An Outdoor Recorded Drone Video Dataset for Action Recognition,”Drones, vol. 3, no. 4, p. 82, 2019
2019
-
[20]
Fed- erated learning in the sky: Aerial-ground air quality sensing framework with uav swarms,
Y . Liu, J. Nie, X. Li, S. H. Ahmed, W. Y . B. Lim, and C. Miao, “Fed- erated learning in the sky: Aerial-ground air quality sensing framework with uav swarms,”IEEE Internet of Things Journal, vol. 8, no. 12, pp. 9827–9837, 2021
2021
-
[21]
Federated learning and autonomous uavs for hazardous zone detection and aqi prediction in iot environment,
P. Chhikara, R. Tekchandani, N. Kumar, M. Guizani, and M. M. Hassan, “Federated learning and autonomous uavs for hazardous zone detection and aqi prediction in iot environment,”IEEE IoT Journal, vol. 8, no. 20, pp. 15 456–15 467, 2021
2021
-
[22]
Communication-Efficient Learning of Deep Networks from Decentral- ized Data,
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentral- ized Data,” inProceedings of the 20th International Conference on Artificial Intelligence and Statistics. PMLR, Apr. 2017, pp. 1273–1282, iSSN: 2640-3498
2017
-
[23]
The MRS UA V System: Pushing the Frontiers of Repro- ducible Research, Real-world Deployment, and Education with Au- tonomous Unmanned Aerial Vehicles,
T. Baca, M. Petrlik, M. Vrba, V . Spurny, R. Penicka, D. Hert, and M. Saska, “The MRS UA V System: Pushing the Frontiers of Repro- ducible Research, Real-world Deployment, and Education with Au- tonomous Unmanned Aerial Vehicles,”J Intell Robot Syst, vol. 102, no. 1, p. 26, Apr. 2021
2021
-
[24]
Flower: A Friendly Federated Learning Research Framework,
D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y . Gao, L. Sani, K. H. Li, T. Parcollet, P. P. B. de Gusm ˜ao, and N. D. Lane, “Flower: A Friendly Federated Learning Research Framework,” Mar. 2022, arXiv:2007.14390 [cs, stat]
2022 arXiv
-
[25]
Scikit-learn: Machine learning in Python,
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,”Journal of Machine L...
2011
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.