REVIEW 4 major objections 8 minor 28 references
Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data
T0 review · 4 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A cloud fuses private robot policies into guide models, and transferring those models back makes imitation learning faster and more accurate without sharing raw data.
desk verdict The framework is clever and coherent, but the reported gains are not cleanly attributed to multi-robot fusion without an ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a cloud knowledge fusion algorithm paired with layer transfer. The cloud maintains an unlabeled database of scenes, each scene captured simultaneously in every sensor modality used by the robots (RGB, depth, semantic segmentation). Each uploaded private model watches the scenes of its own modality and outputs a label suggestion; the median of the private models' outputs becomes the pseudo-label for that scene. The cloud then trains a separate guide model for each sensor type, so a requesting robot receives a model matched to its own sensor. During transfer learning, the robot copies the guide model, freezes its early convolutional feature-extraction layers, and retrains the fully connected layers on local data, which is what gives the faster convergence and lower error rates reported.
What would settle it
Measure the cloud-generated pseudo-labels against human labels on the same held-out scenes; if the median of private model outputs is consistently less accurate than the best individual private model, the knowledge fusion step degrades instead of improving the guide model. A second test would compare FIL controllers with from-scratch controllers in a held-out environment and check whether the FIL controllers actually match or beat them.
Extended reading notes
Core claim
The central claim is that imitation learning in cloud robotic systems can be made more accurate and more sample-efficient without sharing raw data and without requiring the robots to have the same sensor types. The paper argues that a cloud-side guide model, produced by fusing private models through an unsupervised labeling step, carries knowledge that a local robot can absorb by transfer learning. In the experiments, cloud-generated controllers for RGB, depth, and segmentation inputs beat the corresponding local controllers on obstacle hits, missed turns, and straight-driving mistakes; for RGB the cloud controller reduced obstacle hits from 3.45% to 0.69% and missed turns from 12% to 0. Transferred controllers also outperformed generally trained controllers in normal driving and in rain, snow, fog, and dust, and their validation loss started lower and decreased faster during training.
Load-bearing premise
The cloud must have synchronized unlabeled sensor data of the same scenes in every modality used by the local robots (e.g., RGB, depth, and semantic segmentation), so that each private model can label the same scene and the median of those outputs can serve as the training label.
Editorial extensions
If this is right
- A new robot with any supported sensor type can start from the corresponding cloud guide model instead of random weights, lowering the initial loss and the number of training steps needed.
- Robots with different sensor suites (RGB, depth, segmentation) can contribute knowledge to and benefit from one shared skill base without any robot seeing another's raw data.
- The cloud guide model behaves as a cautious aggregate policy: because it is trained on median pseudo-labels, it avoids the extreme steering errors that single-robot policies make at hard turns.
- The transfer scheme improves generalization to unseen conditions: transferred controllers maintained lower error rates than general controllers in rain, snow, fog, and dust in the reported driving simulations.
- Because only network parameters are uploaded, the framework can run asynchronously and does not require the tight synchronization or gradient sharing of typical distributed training.
Reading between the lines
- Editorial inference: The median-based fusion step implicitly assumes that the majority of private models are reliable; a cloud fleet in which most robots share the same bias would produce a guide model with that same bias, and the pseudo-labels would not correct it.
- Editorial inference: The scheme is easiest to realize inside a simulator or other setup with aligned multi-modal capture; in a physical deployment without such paired data, the cloud cannot run the labeling step as specified, so the framework's practical reach may depend on simulation.
- Editorial inference: The same fusion-and-transfer pattern could be tested on other label types (continuous velocities, discrete actions, or grasp poses) and other fusion rules (e.g., confidence-weighted averages) since the paper defines the label calculation as application-dependent.
- Editorial inference: Because the guide model is trained on pseudo-labels, its quality is bounded by the collective quality of the private models; a single expert robot's knowledge could be diluted when fused with many weaker models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Federated Imitation Learning (FIL), a cloud-robotic framework in which local robots train private policies via imitation learning on heterogeneous sensor modalities (RGB, depth, and semantic segmentation), upload only network parameters to a cloud, and the cloud fuses these private models by using them to label unlabeled multi-modal scenes and training a cloud policy on the median of their outputs. This cloud policy is then transferred to a requesting robot by freezing front layers and fine-tuning the remaining layers on local data. Experiments in AirSim/CARLA claim that cloud policies are more accurate than the original local policies and that transferred policies outperform from-scratch policies in a new environment and in adverse weather.
Significance. If the multi-robot fusion effect were cleanly isolated, FIL would be a useful contribution to cloud robotics: it addresses heterogeneous sensor data and avoids raw-data sharing, and the idea of using a simulator to obtain synchronized multi-modal views of the same scenes is a concrete and practical way to enable federated-style model fusion. The transfer experiments across weather conditions provide some out-of-distribution evidence beyond a single environment, and the paper makes its framework and results available. However, the experimental design as reported does not separate the effect of multi-robot knowledge fusion from the effect of training on additional cloud scenes, and the statistical characterization is too weak to support the quantitative accuracy claims. The framework's scope is also narrower than stated because the fusion step requires synchronized multi-modal data that is only available in a simulator or specially instrumented fleet.
major comments (4)
- [Section IV.C, Figs. 8–10, Table 2] The headline claim that FIL improves imitation learning through knowledge from other robots is not isolated. The transferred policy is initialized from the cloud guide model, which was trained on a large corpus of cloud scenes labeled by the private models (Section IV.A, Fig. 4b), whereas the general policy is trained from scratch on local data only. These two conditions differ both in initialization and in the total amount and scope of data seen by the model, so the measured gains could come entirely from generic pre-training on additional scenes, with no contribution from the median-fusion mechanism in Eq. (4). I request an ablation in which the cloud model is trained on the same cloud scenes but with labels from a single private model, or from a single robot's data, with the cloud data held fixed, and then transferred. Without this ablation, the causal claim about "knowledge from other robots" is unsupported.
- [Section IV.B, Tables 1 and 2] The quantitative claims rest on error percentages with no trial counts, confidence intervals, or significance tests. The text says "we conducted 3 experiments" for Table 1 but does not report the per-run results; a single trial change can flip a comparison such as the 16.67% "mistakes in straight" for the RGB local controller. The authors should report the number of evaluation episodes, per-episode errors, and a paired significance test or bootstrap confidence intervals for the differences in Tables 1 and 2. Without this, the conclusion that FIL models "perform significantly better" is not supported by the reported data.
- [Section III.B, Fig. 3] The fusion algorithm requires the cloud to possess synchronized unlabeled sensor data of the same scenes in every modality used by the local robots. As stated in Section III.B, "Before this, it is required to collect different types of sensor data that depends on the types of sensor data in local robots." This precondition is only naturally satisfied in a simulator or a specially instrumented fleet; a general cloud robotic system will not have such paired multi-modal data. The paper should state this as a scope condition and discuss how FIL would be instantiated without a simulation platform. As written, the claimed suitability for cloud robotic systems is broader than the algorithm's actual precondition.
- [Section III.B, Eq. (4), and Section IV.B] The evaluation of cloud policies in Table 1 is partly self-referential: the labels used to train the cloud policy are the median of the outputs of the very local policies with which the cloud policy is compared. Superior accuracy of the cloud policy therefore reflects an ensemble/median-smoothing effect plus access to more scenes, rather than independently validated driving skill. The paper should state this explicitly and provide at least one comparison against a cloud policy trained directly on human labels for a subset of cloud scenes, to validate that the fused labels are not merely averaging out errors of the private models.
minor comments (8)
- [Section III.A] In the framework description, "Similar processes occur in agent A and agent B" should refer to agents B and C.
- [Section III.B] Equations (1)–(5) contain typos and notational inconsistencies: the empirical risk is written as "Remp_Di" and the structural risk as "Rstruct_D" with inconsistent subscripts, and the notation "lin" is ambiguous. Algorithm 1 also contains a non-assignment line "θnt← robotn performs imitation learning".
- [Section IV.B] The text says the third company's policy is based on infrared images, while the figures and the rest of the paper say semantic segmentation images; this inconsistency should be corrected.
- [Table 2] The claim of "significantly better" accuracy is not uniformly supported by the table: for depth images in rain, snow, and fog, and for segmentation images in normal, rain, and snow, the transferred and general policies have equal error rates. The narrative should be tempered or the results reported with uncertainty estimates.
- [Section IV.C, Fig. 10] The training-process comparison plots have unlabeled axes, and the terms "standard_val_loss" and "transferred_val_loss" are not defined; the number of epochs and any early-stopping criterion should also be reported.
- [Section II.C, reference [26]] The statement that federated learning "was first proposed in [26]" is incorrect; [26] is a privacy-attack paper on federated learning, not the original introduction of federated learning, and a proper citation (e.g., McMahan et al.) should be used.
- [Section II.B] The claim that the paper presents "the first privacy-considered knowledge sharing approach for cloud robotic systems" is very strong and should be checked against the related literature; at minimum, a comparison with standard federated averaging in the same setting would help position the contribution.
- [Section IV.A] The network architecture is described only as "similar to VGG-16"; the input size, normalization, optimizer, learning rate, batch size, dataset sizes, and train/validation splits are not reported, which makes the experiments difficult to reproduce.
Circularity Check
Cloud-policy superiority in Table 1 partly reduces to the median-ensemble construction; the transfer claim retains independent content.
-
fitted input called prediction
[Section III.B, Eq. (4)-(5); Section IV.B, Table 1]
"Mein = Median (lin) ... θ∗i(cloud ) = argminθ 1/M ∑M n=1 L( Mein, f(scene(n)in ; θ)) + 1/2 λ‖θ‖2 ... The results are summarized in Table 1. It can be seen from the experimental results that the cloud knowledge improves the local controller that is trained using general imitation learning."
The cloud model's training labels are defined by Eq. (4) as the median of the private models' outputs, and Eq. (5) fits the cloud policy to those very labels. Consequently, the superiority of cloud policies over individual local policies reported in Table 1 is baked into the construction: the cloud policy is a distilled ensemble of the same private policies it is compared against, trained on a larger cloud database. The experiment therefore does not independently validate 'knowledge from other robots'; it demonstrates median aggregation plus extra cloud data. The transfer results in Table 2 are more independent because they test fine-tuned policies in a new neighborhood and weather conditions, but the fusion mechanism itself is not isolated by an ablation.
full rationale
The paper's central claim that FIL improves local imitation learning is not fully circular: the transfer experiments (Table 2, Fig. 10) compare fine-tuned cloud-initialized policies against from-scratch policies in a new neighborhood and under different weather conditions, which provides independent evidence that the guide model carries useful prior knowledge. The self-citation [4] is not load-bearing; it merely points to the authors' prior cloud-robotics work in the introduction. However, the first evaluation pillar (Table 1) is partially circular by construction. The cloud labels are defined as the median of the private-model outputs (Eq. 4), and the cloud policies are trained to reproduce those labels (Eq. 5). Showing that these cloud policies outperform individual local policies therefore largely restates the well-known ensemble/median effect rather than demonstrating a separate 'knowledge from other robots' mechanism. The paper acknowledges that the label calculation 'can draw on some methods of ensemble learning,' but it still presents the resulting superiority as validation of the fusion algorithm. This is a partial reduction of one evaluation claim to the method's own construction, not a complete circularity of the whole framework.
Assumptions & free parameters
free parameters (3)
- L2 regularization coefficient λ =
not reported
- cloud update frequency f =
not reported
- number of frozen layers in transfer =
not reported
assumptions (4)
- domain assumption The cloud possesses unlabeled multi-sensor data (RGB, depth, semantic segmentation) of the same scenes, collected simultaneously.
- domain assumption Sharing only model parameters, never raw data, preserves privacy.
- domain assumption The median of private model outputs is a reliable label for each cloud scene.
- domain assumption Behavioral cloning via supervised learning is an adequate model for imitation.
Cite this review
Pith. "Pith review of Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data." pith.science (2026). https://pith.science/paper/VDKOZQ3W
@misc{pith2026190900895,
author = {Pith},
title = {Pith review of: Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/VDKOZQ3W}},
note = {Machine review of arXiv:1909.00895}
}
read the original abstract
Humans are capable of learning a new behavior by observing others perform the skill. Robots can also implement this by imitation learning. Furthermore, if with external guidance, humans will master the new behavior more efficiently. So how can robots implement this? To address the issue, we present Federated Imitation Learning (FIL) in the paper. Firstly, a knowledge fusion algorithm deployed on the cloud for fusing knowledge from local robots is presented. Then, effective transfer learning methods in FIL are introduced. With FIL, a robot is capable of utilizing knowledge from other robots to increase its imitation learning. FIL considers information privacy and data heterogeneity when robots share knowledge. It is suitable to be deployed in cloud robotic systems. Finally, we conduct experiments of a simplified self-driving task for robots (cars). The experimental results demonstrate that FIL is capable of increasing imitation learning of local robots in cloud robotic systems.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Learning to select and generalize striking movements in robot table tennis,
K. Mlling, J. Kober, O. Kroemer, and J. Peters, “Learning to select and generalize striking movements in robot table tennis,” The International Journal of Robotics Research, vol. 32, no. 3, pp. 263-279, 2013
work page 2013
-
[2]
Towards robust skill generalization: Unifying learning from demonstration and motion planning,
M. Rana, M. Mukadam, S. R. Ahmadzadeh, S. Chernova, and B. Boots, “Towards robust skill generalization: Unifying learning from demonstration and motion planning,” in International Conference on Robot Learning, 2018
work page 2018
-
[3]
Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,
T. Zhang et al., “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 18
work page 2018
-
[4]
B. Liu, L. Wang, M. Liu, and C. Xu, “Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems,” arXiv preprint arXiv:1901.06455, 2019
arXiv 1901
-
[5]
Recent Advances in Imitation Learning from Observation,
F. Torabi, G. Warnell, and P. Stone, “Recent Advances in Imitation Learning from Observation,” arXiv preprint arXiv:1905.13566, 2019
arXiv 1905
-
[6]
A machine learning approach to visual perception of forest trails for mobile robots,
A. Giusti et al., “A machine learning approach to visual perception of forest trails for mobile robots,” IEEE Robotics and Automation Letters (RA-L), vol. 1, no. 2, pp. 661667, 2015
work page 2015
-
[7]
Model-based imitation learning by probabilistic trajectory matching,
P. Englert, A. Paraschos, J. Peters, and M. P. Deisenroth, “Model-based imitation learning by probabilistic trajectory matching,” in IEEE Inter- national Conference on Robotics and Automation, 2013, pp. 19221927
work page 2013
-
[8]
Query-efficient imitation learning for end-to-end simulated driving,
J. Zhang and K. Cho, “Query-efficient imitation learning for end-to-end simulated driving,” in AAAI, 2017
work page 2017
Show all 28 references
-
[9]
End- to-end driving via conditional imitation learning,
F. Codevilla, M. Miiller, A. Lpez, V . Koltun, and A. Dosovitskiy, “End- to-end driving via conditional imitation learning,” in IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 19
2018
-
[10]
End to end learning for self-driving cars,
M. Bojarski et al., “End to end learning for self-driving cars,” arXiv preprint arXiv:1604.07316, 2016
2016 arXiv
-
[11]
Teaching a Robot to Grasp Real Fish by Imitation Learning from a Human Supervisor in Virtual Reality,
Jonatan S. Dyrstad, et al. “Teaching a Robot to Grasp Real Fish by Imitation Learning from a Human Supervisor in Virtual Reality,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2018: 7185-7192
2018
-
[12]
Alvinn: An autonomous land vehicle in a neural network,
D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Advances in neural information processing systems, 1989, pp. 305313
1989
-
[13]
Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty- aware Perspective
L. T, P. Yun, Y . Chen, C. Liu, and H. Y . M. Liu, “Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty- aware Perspective.” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2019
2019
-
[14]
One-shot imitation from observing humans via domain- adaptive meta-learning,
T. Yu et al., “One-shot imitation from observing humans via domain- adaptive meta-learning,” arXiv Prepr. arXiv1802.01557, 2018
2018 arXiv
-
[15]
Cloud-enabled robots,
J. J. Kuffner, “Cloud-enabled robots,” in IEEE-RAS International Con- ference on Humanoid Robotics, Nashville, TN, 2010
2010
-
[16]
A hierarchical auction-based mechanism for real-time resource allocation in cloud robotic systems,
L. Wang, M. Liu, Q. Meng. “A hierarchical auction-based mechanism for real-time resource allocation in cloud robotic systems,” IEEE trans- actions on cybernetics, vol. 47, no. 2, pp.473-484
-
[17]
Cloud robotics: A software architecture: For heterogeneous large-scale autonomous robots,
S. A. Miratabzadeh et al., “Cloud robotics: A software architecture: For heterogeneous large-scale autonomous robots,” in 2016 World Automation Congress (W AC), 2016, pp. 16
2016
-
[18]
Mohanarajah, V
G. Mohanarajah, V . Usenko, M. Singh, et al., Cloud-based collaborative 3D mapping in real-time with low-cost robots,” IEEE Trans. Autom. Sci. Eng., vol. 12, no. 2, pp. 423431, 2015
2015
-
[19]
C2tam: A cloud framework for cooperative tracking and mapping. Robotics and Autonomous Systems,
Luis Riazuelo, Javier Civera, et al. “C2tam: A cloud framework for cooperative tracking and mapping. Robotics and Autonomous Systems,” vol. 62, no.4, pp.401 413, 2014
2014
-
[20]
A tradeoff analysis of a cloud-based robot navigation assistant using stereo image processing,
Javier Salmeron-Garc, Pablo Inigo-Blasco, et al. “A tradeoff analysis of a cloud-based robot navigation assistant using stereo image processing,” IEEE Transactions on Automation Science and Engineering, V ol. 12, No. 2, pp. 444454, 2015
2015
-
[21]
Cloud-based robot grasping with the google object recognition engine,
Ben Kehoe, Akihiro Matsukawa, Sal Candido, James Kuffner, and Ken Goldberg. “Cloud-based robot grasping with the google object recognition engine,” In IEEE International Conference on Robotics and Automation (ICRA), pp. 42634270, 2013
2013
-
[22]
Cloud-based networked visual servo control,
Haiyan Wu, Lei Lou, Chih-Chung Chen, Sandra Hirche, and Kolja Kuhnlenz. “Cloud-based networked visual servo control,” in IEEE Transactions on Industrial Electronics, V ol. 60, No. 2, pp. 554 - 566, 2013
2013
-
[23]
Rospeex: A cloud robotics platform for human-robot spoken dialogues,
Komei Sugiura and Koji Zettsu. “Rospeex: A cloud robotics platform for human-robot spoken dialogues,” in IEEE International Conference on Intelligent Robots and Systems (IROS), pp.6155-6160, 2015
2015
-
[24]
Real-Time Multisensor Data Retrieval for Cloud Robotic Systems,
Wang L , Liu M , Meng Q H. “Real-Time Multisensor Data Retrieval for Cloud Robotic Systems,” IEEE Transactions on Automation Science and Engineering, vol. 12, no. 2, pp. 507-518, 2015
2015
-
[25]
Network offloading poli- cies for cloud robotics: a learning-based approach,
Chinchali S, Sharma A, Harrison J, et al. “Network offloading poli- cies for cloud robotics: a learning-based approach,” arXiv preprint arXiv:1902.05703, 2019
1902 arXiv
-
[26]
Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks,
M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks,” in IEEE Symposium on Security and Privacy, 2018, pp. 1-15
2018
-
[27]
Federated Learning for Mobile Keyboard Prediction,
A. Hard et al., “Federated Learning for Mobile Keyboard Prediction,” arXiv preprint arXiv:1811.03604, 2018
2018 arXiv
-
[28]
In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communica- tion by Federated Learning,
X. Wang, Y . Han, C. Wang, Q. Zhao, X. Chen, and M. Chen, “In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communica- tion by Federated Learning,” arXiv preprint arXiv:1809.07857, 2018
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.