Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Experience Reuse with Probabilistic Movement Primitives

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Initializing policy search with similar movement primitives reduces required learning iterations by more than 60% and improves final skill quality.

desk verdict A usefully honest empirical comparison of mean-only versus full-covariance ProMP initialization for REPS, whose headline 60% speed-up claim is held hostage by an unreported hand-tuned scaling parameter. read the letter →

arxiv 1908.03936 v2 pith:FFXQ2HKA submitted 2019-08-11 cs.LG cs.ROstat.ML

Svenja Stark , Jan Peters , Elmar Rueckert This is my paper · ORCID
classification cs.LGcs.ROstat.ML
keywords transferlearningprobabilisticmovementprimitivesrelativeentropypolicysearchinitializationrobotskillk-nearestneighborslifelongobjectpushing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a robot learning a new motor skill should start from the solutions of similar skills already in its library rather than from scratch. It combines probabilistic movement primitives, which represent a skill as a Gaussian over trajectory parameters, with a description of each skill's effect, so the most relevant past skills can be selected by comparing task descriptors. The selected skill parameters initialize the search distribution of relative entropy policy search, and the paper reports that this initialization improves the final quality of the learned pushing skill and reduces the required learning iterations by more than 60% on a simulated 3-DOF planar arm. Two transfer modes are compared: transferring only the source mean, which is robust and reaches a higher final reward, and transferring the full source covariance, which converges fastest and keeps movements similar to known safe ones but depends on having several good source skills.

What carries the argument

The load-bearing objects are Probabilistic Movement Primitives (ProMPs), a skill stored as a Gaussian over the weights of shared basis functions, giving both a mean trajectory and a trajectory covariance, and Relative Entropy Policy Search (REPS), a sample-based optimizer that repeatedly fits a Gaussian to reward-weighted samples while limiting the KL divergence from its previous distribution. The framework adds a task descriptor to each ProMP and uses Euclidean distance between descriptors with k-nearest neighbors to select source skills. The transfer step itself is the mechanism: the source mean (partial transfer) or the source mean plus scaled covariance (full transfer) is inserted as REPS's initial search distribution, and a hand-tuned scaling factor s keeps that distribution from being too restrictive.

What would settle it

Sweep the scaling factor s over a wide range for the full-transfer mode with k=1 on the harder starting-pose dataset and count iterations to a fixed reward threshold. If no value of s reaches the random baseline's final reward, the claimed more-than-60% iteration saving fails for exactly the setting the paper itself flags as problematic.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the initial distribution of a policy search matters as much as the search algorithm itself, and that a probabilistic skill library can supply that initialization. New skills are learned by taking the mean (partial transfer) or the mean and scaled covariance (full transfer) of the k nearest source skills, chosen by Euclidean distance between task descriptors, and using that as the start of episode-based REPS. Across ten simulated object-pushing tasks, both modes beat a random initialization on final reward in most settings, and full transfer converges within about 25 iterations, saving roughly 60% of samples compared with the baseline's approximately 60 iterations. Full transfer also preserves the qualitative shape of the source trajectories, which the authors identify with safer, more predictable exploration; however, with a single forced nearest neighbor it can get trapped in a local optimum and underperform the baseline, so the benefit is conditional on source quality and library size.

Load-bearing premise

The reported savings rest on a single manually tuned number s that controls how wide the initial search is; if that number must be retuned for each task or library, the full-transfer speed-up is not a robust property of the method.

Editorial extensions

If this is right

  • Full transfer with a well-stocked library can reach the final reward in about 25 iterations instead of the baseline's roughly 60, saving about 60% of the required samples.
  • Learning from the mean of the closest known skills (partial transfer) reliably outperforms random initialization in final reward regardless of library size, making it the safer default when similar source skills are scarce.
  • Full transfer preserves the shape of the known movement, so exploration stays near previously safe trajectories, at the cost of sometimes settling for a lower reward than partial transfer reaches.
  • Transferring the full covariance only pays off when several similar source skills are available; with a single forced source it can get stuck in a local optimum and fall below the random baseline.
  • Because REPS treats the reward as a black box, the framework needs no model of the object–arm contact dynamics to benefit from the initialization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the iteration saving transfers to other policy-search algorithms; a natural check would be to replace REPS with any KL-regularized distribution-based optimizer and see whether the saving persists.
  • The k=1 full-transfer failure suggests a safeguard the paper does not implement: monitor the KL divergence between the source distribution and the REPS updates, and switch to partial transfer when the search distribution stays pinned to the source.
  • For a lifelong-learning curriculum, the result implies an ordering rule: learn a cluster of similar tasks before a distant one, because full transfer is reliable only when the library contains several comparable source skills.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents a framework for reusing previously learned robot movement skills when learning a new task. Skills are represented as probabilistic movement primitives (ProMPs), and relative entropy policy search (REPS) is used to adapt an initial distribution. The authors propose two transfer modes: partial transfer, which copies the mean of selected source skill parameters and uses an isotropic covariance sI, and full transfer, which also copies a scaled source covariance. The methods are evaluated on a simulated 3-DOF planar arm pushing a cylinder, comparing against a random initialization. The reported results show faster convergence and higher final reward for most transfer settings, and the paper claims a reduction of more than 60% in required iterations.

Significance. The paper addresses a relevant problem in robot learning: reusing skill libraries to speed up policy search. The combination of ProMPs with REPS is natural and the qualitative analysis of trajectory similarity (Figure 5) is informative. The authors honestly report the failure of full transfer with k=1 on dataset B (Section IV-B4), which is a strength of the exposition. If the empirical claims are made robust, the framework could be a practical component for lifelong robot learning. However, the current support for the headline speed-up claim is insufficient.

major comments (4)
  1. [Section III-C and Section IV-B4] The headline claim of a reduction of more than 60% in required iterations rests on the full-transfer mode, whose covariance is scaled by the hand-tuned factor s as s/max(Σk)Σk. The paper never reports the value of s, provides no selection rule, and gives no sensitivity analysis. Moreover, Section IV-B4 states that increasing s did not resolve the k=1 full-transfer failure on dataset B, indicating that the interaction between s and the source covariance, rather than transfer per se, determines the outcome. A sweep over s (or a principled way to fix it) is necessary to establish that the speed-up is a property of the method and not an artifact of the chosen scaling.
  2. [Section IV-B1 and Conclusion] The claim of 'more than 60%' sample reduction is not precisely quantified and appears inconsistent with the reported numbers. The text states that full transfer converges within 'latest 25 iterations' while the baseline converges around iteration 60, which corresponds to roughly 58% reduction, not more than 60%. No definition of 'required iterations' (for example, the iteration at which reward reaches a threshold or a stopping criterion) is given, and no statistical test or confidence interval is provided for the comparison. The abstract's unqualified statement that 'required iterations ... can be reduced by more than 60%' is therefore not supported by the evidence.
  3. [Section III-B and Section IV-A3] The REPS KL constraint epsilon is a free parameter that directly controls the step size and thus the convergence speed; its value is never reported. Similarly, the reward weights a and b are hand-tuned (Section IV-A3) and their values are not given. Since the evaluation compares reward curves across settings, the omission of these parameters makes the experiments irreproducible and prevents assessing whether the observed speed-ups are specific to the chosen hyperparameters. Please report all hyperparameters and, where feasible, provide a sensitivity analysis.
  4. [Abstract and Section IV-B4] The abstract claims that 'the quality of the learned skills improves and the required iterations ... can be reduced by more than 60% when past experiences are utilized.' Yet the paper's own results show that the full-transfer mode with k=1 on dataset B fails to outperform the baseline, and the partial-transfer mode does not carry the 60% speed-up. The conclusion restricts the 60% figure to 'a well initialized full transfer,' but the abstract is unqualified. The empirical claims need to be scoped to the settings in which they hold.
minor comments (6)
  1. [Section IV-A3] The description of the reward weights says a and b are tuned so that the mean of rT is 1.5 times the mean of rp. Please report the actual values and the procedure used to estimate these means, as this affects the reward scale in Figure 2.
  2. [Section III-C] The baseline initialization is defined as µb = λµN + (1−λ)µr with λ based on the basis functions at time zero. The rationale for this particular baseline should be explained more clearly, since it may already contain a form of prior knowledge through µN.
  3. [Section IV-B] The sentence 'Each of the transfer possibilities is evaluated five times. This results in overall 5×460 = 2300 evaluations' does not explain the origin of the factor 460. Please clarify how many transfer possibilities are counted.
  4. [Section IV-B1] The sentence 'The convergence takes approximately 60 iterations, which is about the same number of iterations as the baseline' is ambiguous because the antecedent is the partial transfer with k=1. Please rephrase to clarify.
  5. [Figure 5] The lower row would benefit from a caption explaining what the yellow bars and red dots represent in more detail, especially for readers not familiar with ProMP weight visualization.
  6. [Section II] The distance measure from [30] is cited to the authors' own earlier work; adding a one-sentence description of the measure would make the paper self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the reported speed-up is an empirical measurement and no derivation step reduces to its inputs.

full rationale

The paper's central claim, a more-than-60% reduction in required learning iterations, is an empirical comparison of reward curves for different initialization strategies (Figure 2, Section IV-B). The REPS update is a standard weighted maximum-likelihood refinement of the initial distribution q(τ), and the improvement is measured against a baseline, not derived from a fitted parameter. The main potential points of concern are the hand-tuned covariance scaling factor s (Section III-C) and the self-citation [30] for Euclidean distance as a task-descriptor similarity measure. Neither is circular: s is an experimental tuning parameter whose sensitivity is explicitly acknowledged and probed (Section IV-B4 reports that increasing s did not cure the k=1 full-transfer failure), and the [30] citation is used only to justify a standard distance measure for selecting source skills, not to define the performance gain. No equation in the paper defines the predicted speed-up in terms of the transferred parameters, the scaling factor, or the self-cited distance result. The reported failure cases and robustness limitations are correctness concerns, not circularity. The derivation chain is therefore self-contained and externally falsifiable.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The core contribution is an empirical framework built on existing algorithms; the central claim rests on four domain assumptions and several unreported hand-tuned constants, with no new invented entities.

free parameters (3)
  • covariance scaling factor s
    Hand-tuned in Section III-C and used in both transfer modes; no value or tuning procedure is reported, yet full transfer performance is sensitive to it.
  • reward weighting factors a and b
    Hand-tuned in Section IV-A.3 so the object-trajectory term has 1.5 times the mean magnitude of the pushing term; exact values are not given.
  • REPS KL constraint epsilon
    Controls the step size of the policy update in Section III-B; not specified in the paper, so replication requires choosing it.
assumptions (4)
  • domain assumption Euclidean distance between task descriptors is a sufficient similarity measure for selecting useful source skills.
    Section IV-A.2 defines D_Ti = ||T* - Ti||_2 and uses k-NN; the paper cites prior work [30] but does not validate that this distance predicts transfer benefit for the tested pushing tasks.
  • domain assumption Target task descriptors T* are provided externally by a planner or human.
    Section I states this assumption explicitly; the robot cannot infer the goal on its own, and the whole selection mechanism depends on having T*.
  • domain assumption Open-loop ProMP parameterization with six Gaussian basis functions per joint is expressive enough to represent successful pushing trajectories.
    Section IV-A sets 18 trajectory parameters; if this representation is too restrictive, no initialization choice could produce the claimed improvements.
  • domain assumption PyBullet simulation dynamics faithfully capture the contact-rich object pushing behavior relevant to the claim.
    All experiments are simulated (Section IV-A); no real robot validation is reported, so safety and sample-saving claims depend on simulator fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Experience Reuse with Probabilistic Movement Primitives." pith.science (2026). https://pith.science/paper/FFXQ2HKA

@misc{pith2026190803936,
  author       = {Pith},
  title        = {Pith review of: Experience Reuse with Probabilistic Movement Primitives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFXQ2HKA}},
  note         = {Machine review of arXiv:1908.03936}
}
read the original abstract

Acquiring new robot motor skills is cumbersome, as learning a skill from scratch and without prior knowledge requires the exploration of a large space of motor configurations. Accordingly, for learning a new task, time could be saved by restricting the parameter search space by initializing it with the solution of a similar task. We present a framework which is able of such knowledge transfer from already learned movement skills to a new learning task. The framework combines probabilistic movement primitives with descriptions of their effects for skill representation. New skills are first initialized with parameters inferred from related movement primitives and thereafter adapted to the new task through relative entropy policy search. We compare two different transfer approaches to initialize the search space distribution with data of known skills with a similar effect. We show the different benefits of the two knowledge transfer approaches on an object pushing task for a simulated 3-DOF robot. We can show that the quality of the learned skills improves and the required iterations to learn a new task can be reduced by more than 60% when past experiences are utilized.

Figures

Figures reproduced from arXiv: 1908.03936 by the authors.

Figure 1
Figure 1. An exemplary depiction of the 2D learning scenario which is used [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Mean reward per iteration for learning on a large library, averaged over the learning of all ten different skills. The source knowledge is selected [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Mean reward per iteration for learning on a small library, averaged over the learning of all ten different skills. The source knowledge is selected [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Mean reward for each iteration, averaged over the learning of all ten different skills. The source knowledge is selected from all possible skill [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: To grasp the qualitative results of the transfer learning approaches, we show exemplary resulting behavior of five learning trials for each of the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 31 canonical work pages

  1. [1]

    The development of motor behavior,

    K. E. Adolph and J. M. Franchak, “The development of motor behavior,” Wiley Interdisciplinary Reviews: Cognitive Science , 2017

    work page 2017

  2. [2]

    Probabilistic movement primitives,

    A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann, “Probabilistic movement primitives,” in Advances in Neural Information Processing Systems 26, 2013

    work page 2013

  3. [3]

    Transfer learning for reinforcement learning domains: A survey,

    M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” The Journal of Machine Learning Research , 2009

    work page 2009

  4. [4]

    A survey of transfer learning,

    K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” Journal of Big Data , 2016

    work page 2016

  5. [5]

    A survey on transfer learning,

    S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering , 2010

    2010

  6. [6]

    Neuroscience-inspired artificial intelligence,

    D. Hassabis, D. Kumaran, C. Summerfield, and M. Botvinick, “Neuroscience-inspired artificial intelligence,” Neuron, 2017

    work page 2017

  7. [7]

    A survey on deep transfer learning,

    C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in International conference on artificial neural networks, Springer, 2018

    work page 2018

  8. [8]

    Building machines that learn and think like people,

    B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, “Building machines that learn and think like people,” Behavioral and brain sciences, 2017

    work page 2017

Show all 32 references
  1. [9]

    Nips 2016 tutorial: Nuts and bolts of building ai applications using deep learning by andrew ng

    B.-H. Kim, “Nips 2016 tutorial: Nuts and bolts of building ai applications using deep learning by andrew ng.” https://www.youtube.com/watch?v=wjqaz6m42wU. 2019-02-13

    2016

  2. [10]

    Multi-robot transfer learning: A dynamical system perspective,

    M. K. Helwa and A. P. Schoellig, “Multi-robot transfer learning: A dynamical system perspective,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , IEEE, 2017

    2017

  3. [11]

    Accelerating model learning with inter-robot knowledge transfer,

    N. Makondo, B. Rosman, and O. Hasegawa, “Accelerating model learning with inter-robot knowledge transfer,” 2018 IEEE International Conference on Robotics and Automation (ICRA) , 2018

    2018

  4. [12]

    Sim-to-real transfer learning using robustified controllers in robotic tasks involving complex dynamics,

    J. van Baar, A. Sullivan, R. Cordorel, D. Jha, D. Romeres, and D. Nikovski, “Sim-to-real transfer learning using robustified controllers in robotic tasks involving complex dynamics,” in 2019 International Conference on Robotics and Automation (ICRA) , IEEE, 2019

    2019

  5. [13]

    Zero shot transfer learning for robot soccer,

    D. Schwab, Y . Zhu, and M. M. Veloso, “Zero shot transfer learning for robot soccer,” in International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , 2018

    2018

  6. [14]

    Transfer learning for reinforce- ment learning on a physical robot,

    S. Barrett, M. E. Taylor, and P. Stone, “Transfer learning for reinforce- ment learning on a physical robot,” in Ninth International Conference on Autonomous Agents and Multiagent Systems - Adaptive Learning Agents Workshop (AAMAS - ALA) , 2010

    2010

  7. [15]

    Probabilistic policy reuse in a reinforce- ment learning agent,

    F. Fern´andez and M. Veloso, “Probabilistic policy reuse in a reinforce- ment learning agent,” in Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS) , 2006

    2006

  8. [16]

    Transfer learning,

    L. Torrey and J. Shavlik, “Transfer learning,” in Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, 2010

    2010

  9. [17]

    Transfer learning for policy search methods,

    M. E. Taylor, S. Whiteson, and P. Stone, “Transfer learning for policy search methods,” in In ICML Workshop on Structural Knowledge Transfer for Machine Learning , 2006

    2006

  10. [18]

    Contextual relative entropy policy search with covariance matrix adaptation,

    A. Abdolmaleki, D. Simes, N. Lau, L. P. Reis, and G. Neumann, “Contextual relative entropy policy search with covariance matrix adaptation,” in 2016 International Conference on Autonomous Robot Systems and Competitions (ICARSC) , 2016

    2016

  11. [19]

    Using proba- bilistic movement primitives in robotics,

    A. Paraschos, C. Daniel, J. Peters, and G. Neumann, “Using proba- bilistic movement primitives in robotics,” Autonomous Robots, 2018

    2018

  12. [20]

    Extracting low-dimensional control variables for movement primitives,

    E. Rueckert, J. Mundo, A. Paraschos, J. Peters, and G. Neumann, “Extracting low-dimensional control variables for movement primitives,” in Proceedings of the International Conference on Robotics and Automation (ICRA), 2015

    2015

  13. [21]

    Probabilistic movement models show that postural control precedes and predicts volitional motor control,

    E. Rueckert, J. Camernik, J. Peters, and J. Babic, “Probabilistic movement models show that postural control precedes and predicts volitional motor control,” Nature PG: Scientific Reports , 2016

    2016

  14. [22]

    Relative entropy policy search,

    J. Peters, K. M ¨ulling, and Y . Altun, “Relative entropy policy search,” Proceedings of the Twenty-Fourth National Conference on Artificial Intelligence, 2010

    2010

  15. [23]

    A survey on policy search for robotics,

    M. P. Deisenroth, G. Neumann, and J. Peters, “A survey on policy search for robotics,” Foundations and Trends in Robotics , 2013

    2013

  16. [24]

    Natural evolution strategies.,

    D. Wierstra, T. Schaul, T. Glasmachers, Y . Sun, J. Peters, and J. Schmidhuber, “Natural evolution strategies.,” Journal of Machine Learning Research, 2014

    2014

  17. [25]

    Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation,

    N. Hansen and A. Ostermeier, “Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation,” in Proceedings of IEEE international conference on evolutionary computation, 1996

    1996

  18. [26]

    Simple random search of static linear policies is competitive for reinforcement learning,

    H. Mania, A. Guy, and B. Recht, “Simple random search of static linear policies is competitive for reinforcement learning,” in Advances in Neural Information Processing Systems 31 , 2018

    2018

  19. [27]

    An introduction to kernel and nearest-neighbor nonparametric regression,

    N. S. Altman, “An introduction to kernel and nearest-neighbor nonparametric regression,” The American Statistician , 1992

    1992

  20. [28]

    Dynamical movement primitives: Learning attractor models for motor behaviors,

    A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical movement primitives: Learning attractor models for motor behaviors,” Neural Computation, 2013

    2013

  21. [29]

    Pybullet, a python module for physics simulation for games, robotics and machine learning

    E. Coumans and Y . Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning.” http:// pybullet.org, 2016–2018

    2016

  22. [30]

    A comparison of distance measures for learning nonparametric motor skill libraries,

    S. Stark, J. Peters, and E. Rueckert, “A comparison of distance measures for learning nonparametric motor skill libraries,” in Proceedings of the International Conference on Humanoid Robots (HUMANOIDS) , 2017

    2017

  23. [31]

    The similarity paradox in human learning: A resolution,

    C. E. Osgood, “The similarity paradox in human learning: A resolution,” Psychological Review, 1949

    1949

  24. [32]

    Transfer of training: A review of research and practical implications,

    J. Annett and J. Sparrow, “Transfer of training: A review of research and practical implications,” Innovations in Education and Training International, 1985

    1985

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.