Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Learning to Explore in Motion and Interaction Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that replacing random exploration noise in DDPG with samples from a generative model trained on previously solved tasks more than doubles learning speed on new motion and contact tasks, especially when rewards are sparse.

desk verdict A genuinely new exploration mechanism with honest empirical support, but the additive-noise assumption is untested and the evaluation protocol is too loose for the speedup claims as stated. read the letter →

arxiv 1908.03731 v1 pith:KT77R2PN submitted 2019-08-10 cs.RO cs.AIcs.LG

Miroslav Bogdanovic , Ludovic Righetti This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords learnedexplorationgenerativemodelDDPGLSTMsparserewardsrobotmanipulationcontacttaskstransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a robot can learn new tasks faster by reusing the motions it already knows, without reshaping the reward. It replaces the random exploration noise in a standard deep reinforcement learning algorithm for continuous control with samples from an LSTM that generates short action sequences conditioned on recent state history, trained on trajectories from previously solved tasks. In simulated 7-degree-of-freedom manipulator tasks, from reaching to periodic motion with contact force regulation, this learned exploration process more than doubles learning speed compared with standard DDPG using Gaussian or Ornstein-Uhlenbeck noise, and the gain is largest when rewards are sparse. The authors also claim that the method is robust to parameter tuning and can continually update the exploration model as new tasks are solved.

What carries the argument

The Learned Exploration Process (LEP) is a recurrent LSTM network trained by maximum likelihood on short, randomly sampled $h$-step sub-trajectories taken from policies that solved previous tasks. It maps a history of states to the parameters of a diagonal Gaussian distribution over the corresponding action sequence. This machinery carries the argument: conditioning on state histories instead of a single state keeps the combined distribution of previous policies focused, avoiding the uniform and uninformative action distribution that arises when only the current state is used. In the reinforcement learning loop, LEP replaces the independent noise process in DDPG, so each action is the current deterministic policy plus one LEP sample, with the LSTM internal state reset every $h$ steps to match the training sequence length.

What would settle it

Train LEP only on reaching trajectories and measure episodes-to-success on a sparse-reward contact task over many random seeds; if LEP-equipped DDPG does not beat tuned DDPG with Ornstein-Uhlenbeck noise, the transfer claim is not supported. To isolate the mechanism, repeat with LEP trained on random or non-informative trajectories and check that the speedup disappears, which would show the effect comes from task structure rather than from the recurrent network itself.

Watch

Extended reading notes

Core claim

The central discovery is that exploration noise can itself be learned from past experience instead of being sampled from a generic, unstructured process. Conditioning a combined policy distribution on a short history of past states, rather than on the current state alone, concentrates the distribution into behaviors resembling those of the previously learned policies; the paper illustrates this with a point-mass reach task and then instantiates it as the Learned Exploration Process (LEP). In DDPG, where actions are computed as $a_t = \pi(s_t) + \epsilon$, LEP replaces the external noise source $\epsilon$ with a sample from an LSTM-predicted diagonal Gaussian over action sequences, reset every $h$ steps. Across four simulated tasks on a 7-DoF arm, the LEP-equipped DDPG converges at least twice as fast on the sparse-reward contact tasks and produces more consistent final policies, while on a periodic-motion task whose movement profiles differ from the training data it performs no worse than standard DDPG.

Load-bearing premise

The load-bearing premise is that short segments of behavior from previously solved tasks contain motion regularities that transfer to new tasks on the same robot, with a brief state history enough to expose them, and that adding these samples to the changing deterministic policy remains sensible throughout training.

Editorial extensions

If this is right

  • New tasks can be learned more than twice as fast when rewards are sparse, provided the required movement profiles resemble those present in the training data.
  • The exploration model can be retrained with data from newly learned tasks, so a curriculum of tasks from simple to complex can progressively enrich the exploration strategy.
  • Because only trajectories are needed to train LEP, the exploration prior can be built from any source of good behavior, including trajectory optimization or demonstration, not just from full policies.
  • The method reduces sensitivity to parameter tuning and random initialization, making final performance more consistent across task instances and hyperparameter settings.
  • The approach is not tied to DDPG: any off-policy algorithm with independent exploration noise could in principle adopt the same learned exploration process.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to vary the history length $h$: the point-mass argument predicts that as $h$ shrinks toward zero the learned exploration distribution should flatten toward the uninformative uniform distribution, so learning curves across $h$ would directly test the proposed mechanism.
  • If the speedup stems from injecting temporally correlated, low-level motion structure, the same principle could transfer to other off-policy algorithms such as soft actor-critic, or to sim-to-real settings where safe motion priors constrain exploration.
  • Since LEP needs only trajectories, a natural next step is to assemble the exploration prior from offline datasets of previously demonstrated or optimized behaviors, potentially removing the need for any prior reinforcement learning on the system.
  • The robustness results suggest that replacing unstructured noise with structured samples reduces variance across random seeds; an experiment comparing seed variance of LEP-DDPG against standard DDPG across many tasks would quantify this effect directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a learned exploration process (LEP) for deep reinforcement learning on robotic manipulation and contact tasks. A recurrent network is trained on short trajectories from previously solved tasks to model action sequences conditioned on state histories, and the resulting samples are added to the deterministic policy of DDPG in place of standard exploration noise (Section II-B, Eq. (5)). The authors report experiments on four simulated KUKA LWR tasks: reaching, stationary force application, periodic motion, and periodic motion with contact force regulation. They claim that the approach more than doubles learning speed relative to standard DDPG and PPO when rewards are sparse, while remaining robust to parameter choices and enabling continuous learning as new tasks are solved.

Significance. If the central claim holds, the paper offers a practical way to reuse trajectory data from prior tasks as an exploration prior without modifying reward functions or task structure. The idea is well motivated by the point-mass example, and the experimental design covers a meaningful progression from free-space reaching to contact-rich periodic motion with sparse rewards. The paper also reports a helpful negative result: on a dense-reward periodic task whose motion profile differs from the training data (Task 3, Figure 5), LEP does not speed up learning but also does not hurt performance. The four-task evaluation, the continuous-learning update of the model, and the robustness check in Figure 7 are strengths. However, the manuscript does not ship code, hyperparameter tables, or seed-level statistics, so the experiments are not fully reproducible from the text alone, and the causal interpretation of the speedup is not yet fully supported.

major comments (3)
  1. [II-B, Eq. (5), Remark 1] The mechanism behind the reported speedups is asserted rather than demonstrated. The LEP is trained to reproduce full action sequences (Section II-A), yet at deployment it is added to the current deterministic policy as u_t = pi(s_t) + epsilon_LEP. Remark 1 concedes that "there is no guarantee that the sum with output of the current policy will do the same," and the only supporting argument is that the policy is initialized near zero. Because the policy changes during training and the tasks involve torque limits and contact forces (Tasks 2 and 4), the composite command can leave the distribution of actions on which the LEP was trained. Without either a distributional analysis of u_t relative to the LEP training data, an ablation that compares LEP against noise of matched amplitude or variance, or a variant that learns residual noise conditioned on pi(s_t), the paper does not establish that the learned motion statistics, rather than the altered noise structure, cause the speedups.
  2. [III-C, IV-A, IV-D] The quantitative speedup claim rests on an evaluation protocol that is not fully specified. The learning curves in Figures 3-6 show the best configuration per algorithm, while Section III-C defines the success threshold empirically from observed behaviors. No significance tests or confidence intervals are reported, and the number of random seeds is not stated despite the paper citing reference [21] on seed variability. Figure 7 shows robustness to parameterization only for Task 4 and not for the other tasks. To support the "more than double learning speed" claim, the authors should report aggregate results across all tested configurations and seeds, state the success-threshold selection rule, and test sensitivity of the measured speedups to the chosen threshold.
  3. [II-C, IV-C] Section II-C describes a continuous learning loop in which the exploration model is repeatedly expanded as new tasks are solved, but Section IV-C tests only a single additional round of training: one LEP model is trained on data from Tasks 2 and 3 and then evaluated on Task 4. The abstract's claim that the approach "enables continuous learning of improved exploration strategies as novel tasks are learned" is therefore not supported by an experiment with multiple expansions. An additional experiment with two or more successive updates of the exploration model would be needed to substantiate this claim.
minor comments (5)
  1. [II-A] The notation s_{1:t} is used in the point-mass example to denote both full trajectories and shorter windows, while the training procedure uses segments of length h. Please clarify the conditioning variable precisely and specify whether the LEP is conditioned on states only or on state-action pairs.
  2. [Figures 3-6] The gray line labeled "Task success" and the vertical dashed lines marking convergence are not defined in the captions. Please state how the success threshold is determined from the reward and how the convergence point is computed from the learning curves.
  3. [III-C, IV-A] The empirical success threshold is described only qualitatively. Please provide a table of the threshold values for all four tasks and justify that the reported speedups are not sensitive to small changes in these thresholds.
  4. [IV-A, IV-C] The number of random seeds used for each learning curve is not reported, and it is unclear whether the "100 policies" used to train LEP are independent training runs or multiple snapshots from a few runs. Please report the number of seeds and the variance across seeds.
  5. [Experimental Setup] No hyperparameter table or code release is provided, which makes the experiments difficult to reproduce. Please include a table with network architectures, learning rates, replay buffer sizes, action noise parameters, and the LEP subsequence lengths tested.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported learning-speed gains are measured against external baselines, and the transfer mechanism is empirically tested rather than derived from its own inputs.

full rationale

The paper's central claim is an empirical speedup in simulated DDPG/PPO comparisons, not a derivation. The exploration model LEP is constructed by maximizing log-likelihood of action sequences from previously solved tasks (Section II-A) and is then added to the DDPG policy in Eq. (5); no equation identifies the new task's reward or measured learning curves with the LEP training objective or with the prior-task data. The success threshold in Section III-C is an evaluation criterion chosen so that partial behaviors do not count as solved, and it does not enter the learning updates or the LEP loss, so it cannot force the relative ordering of the plotted curves. Remark 1 explicitly concedes there is 'no guarantee' that the additive combination remains sensible and justifies the approach by observed practice, which makes the transfer mechanism a tested hypothesis rather than an assumption smuggled in by construction. The post hoc selection of best configurations in Section IV is a reporting choice, and the robustness results in Section IV-D report across all parametrizations. The only self-citation (ref. [16]) concerns trajectory optimization as a possible data source and is not load-bearing. The comparisons use external implementations (OpenAI Baselines) and external algorithms (DDPG, PPO), so the speedup claims are falsifiable measurements. Any concerns about post-hoc threshold selection or the untested additivity assumption bear on soundness and generalization, not on circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

All load-bearing premises are the transferability of short-horizon motion patterns across tasks, the informativeness of state-history conditioning, and the assumption that adding LEP samples to the changing policy remains useful. The free parameters are mostly user-chosen hyperparameters and the empirical success threshold; none are fitted by a derivation, but several are tuned during evaluation.

free parameters (5)
  • Success reward threshold T = not reported
    Defined empirically per task in Section III-C by analyzing behaviors; used to declare tasks solved and to measure convergence speed.
  • Subsequence length h = not reported
    Used to train LEP and to reset the LSTM; selected among tested values; affects the exploration model's conditioning horizon.
  • Number of source policies = 100
    Section IV-C collects 100 policies from each of Tasks 2 and 3; this is a user-chosen quantity.
  • DDPG exploration noise parameters = not reported
    Baseline DDPG was evaluated with Gaussian and OU noise over a range of variances; best configuration selected for comparisons (Section IV-A).
  • LEP network architecture = not reported
    LSTM architecture details (hidden size, depth) are not reported, though training follows [18].
assumptions (4)
  • domain assumption Conditioning on a state history yields a focused action distribution; conditioning on a single state yields a uniform distribution.
    Motivated by a point-mass example in Section II-A; not proven for the actual 7-DoF robot or general control tasks.
  • ad hoc to paper The sum of the deterministic policy output and the LEP sample remains a sensible action.
    Remark 1 explicitly states there is no guarantee; the paper relies on the DDPG policy initializing near zero so early exploration is pure LEP samples.
  • domain assumption Behavioral regularities from previously solved tasks transfer to new tasks on the same robot.
    The entire approach depends on this; Section II frames it as exploiting the same system across tasks.
  • domain assumption Short trajectory segments capture reusable motion primitives rather than task-specific solutions.
    In Section II-A the authors limit training samples to length h to avoid overfitting to individual tasks; this is a modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Explore in Motion and Interaction Tasks." pith.science (2026). https://pith.science/paper/KT77R2PN

@misc{pith2026190803731,
  author       = {Pith},
  title        = {Pith review of: Learning to Explore in Motion and Interaction Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KT77R2PN}},
  note         = {Machine review of arXiv:1908.03731}
}
read the original abstract

Model free reinforcement learning suffers from the high sampling complexity inherent to robotic manipulation or locomotion tasks. Most successful approaches typically use random sampling strategies which leads to slow policy convergence. In this paper we present a novel approach for efficient exploration that leverages previously learned tasks. We exploit the fact that the same system is used across many tasks and build a generative model for exploration based on data from previously solved tasks to improve learning new tasks. The approach also enables continuous learning of improved exploration strategies as novel tasks are learned. Extensive simulations on a robot manipulator performing a variety of motion and contact interaction tasks demonstrate the capabilities of the approach. In particular, our experiments suggest that the exploration strategy can more than double learning speed, especially when rewards are sparse. Moreover, the algorithm is robust to task variations and parameter tuning, making it beneficial for complex robotic problems.

Figures

Figures reproduced from arXiv: 1908.03731 by the authors.

Figure 1
Figure 1. Point mass example illustrating our approach (cf. Section II-A for details): (a) Task: moving from one point to another; [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the four tasks tested in the experiments. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We notice that all the algorithms converge to a good behavior (above the gray line), which is expected for this simple task. PPO, being an on-policy algorithm, converges noticeably slower than the other two. While standard DDPG implementation and the one using our exploration process both converge relatively quickly, the average for our ap￾proach reaches the desired task value more than two times faster than the one… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Learning results of the best performing policies for [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Learning results of the best performing policies for [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Learning results of the best performing policies for [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Robustness to parametrization results for Task 4. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 6 canonical work pages

  1. [21]

    How many random seeds? statistical power analysis in deep reinforcement learning experiments,

    C. Colas, O. Sigaud, and P.-Y . Oudeyer, “How many random seeds? statistical power analysis in deep reinforcement learning experiments,” arXiv preprint arXiv:1806.08295 , 2018

    arXiv 2018

  2. [1]

    Learning agile and dynamic motor skills for legged robots,

    J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V . Tsounis, V . Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, p. eaau5872, 2019

    2019

  3. [2]

    Control policy with autocorrelated noise in reinforce- ment learning for robotics,

    P. Wawrzynski, “Control policy with autocorrelated noise in reinforce- ment learning for robotics,”International Journal of Machine Learning and Computing, vol. 5, no. 2, p. 91, 2015

    work page 2015

  4. [3]

    On the theory of the brownian motion,

    G. E. Uhlenbeck and L. S. Ornstein, “On the theory of the brownian motion,” Phys. Rev. , vol. 36, pp. 823–841, Sep 1930. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRev.36.823

    doi:10.1103/physrev.36.823 1930

  5. [4]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” CoRR, vol. abs/1509.02971, 2015. [Online]. Available: http://arxiv.org/abs/1509.02971

    arXiv 2015

  6. [5]

    Parameter space noise for exploration,

    M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y . Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz, “Parameter space noise for exploration,” CoRR, vol. abs/1706.01905, 2017. [Online]. Available: http://arxiv.org/abs/1706.01905

    arXiv 2017

  7. [6]

    Intrinsically motivated learning of hierarchical collec- tions of skills,

    A. G. Barto, “Intrinsically motivated learning of hierarchical collec- tions of skills,” International Conference on Developmental Learning and Epigenetic Robotic , pp. 112–119, 2004

    work page 2004

  8. [7]

    Curiosity driven exploration of learned disentangled goal spaces,

    A. Laversanne-Finot, A. P ´er´e, and P.-Y . Oudeyer, “Curiosity driven exploration of learned disentangled goal spaces,” arXiv preprint arXiv:1807.01521, 2018

    arXiv 2018

Show all 21 references
  1. [8]

    Curiosity-Driven Exploration by Self-Supervised Prediction,

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-Driven Exploration by Self-Supervised Prediction,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops , vol. 2017-July, pp. 488–489, 2017

    2017

  2. [9]

    Distral: Robust multitask reinforcement learning,

    Y . W. Teh, V . Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu, “Distral: Robust multitask reinforcement learning,” CoRR, vol. abs/1707.04175, 2017. [Online]. Available: http://arxiv.org/abs/1707.04175

    2017 arXiv

  3. [10]

    Learning and transfer of modulated locomotor controllers,

    N. Heess, G. Wayne, Y . Tassa, T. P. Lillicrap, M. A. Riedmiller, and D. Silver, “Learning and transfer of modulated locomotor controllers,” CoRR, vol. abs/1610.05182, 2016. [Online]. Available: http://arxiv.org/abs/1610.05182

    2016 arXiv

  4. [11]

    Learning by playing - solving sparse reward tasks from scratch,

    M. A. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. V . de Wiele, V . Mnih, N. Heess, and J. T. Springenberg, “Learning by playing - solving sparse reward tasks from scratch,” CoRR, vol. abs/1802.10567, 2018. [Online]. Available: http://arxiv.org/abs/1802. 10567

    2018 arXiv

  5. [12]

    Dynamic movement primitives-a framework for motor control in humans and humanoid robotics,

    S. Schaal, “Dynamic movement primitives-a framework for motor control in humans and humanoid robotics,” in Adaptive motion of animals and machines . Springer, 2006, pp. 261–280

    2006

  6. [13]

    Dynamical movement primitives: learning attractor models for motor behaviors,

    A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical movement primitives: learning attractor models for motor behaviors,” Neural computation, vol. 25, no. 2, pp. 328–373, 2013

    2013

  7. [14]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  8. [15]

    Contact-Invariant Opti- mization for Hand Manipulation,

    I. Mordatch, Z. Popovi ´c, and E. Todorov, “Contact-Invariant Opti- mization for Hand Manipulation,” in Eurographics/ ACM SIGGRAPH Symposium on Computer Animation , 2012, pp. 1–8

    2012

  9. [16]

    On Time Optimization of Centroidal Momentum Dynamics,

    B. Ponton, A. Herzog, A. Del Prete, S. Schaal, and L. Righetti, “On Time Optimization of Centroidal Momentum Dynamics,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . Brisbane, Australia: IEEE, May 2018, pp. 5776–5782. [Online]. Available: https://ar...

    2018 arXiv

  10. [17]

    Robot program- ming by demonstration,

    A. Billard, S. Calinon, R. Dillmann, and S. Schaal, “Robot program- ming by demonstration,” Springer handbook of robotics , pp. 1371– 1394, 2008

    2008

  11. [18]

    Generating sequences with recurrent neural networks,

    A. Graves, “Generating sequences with recurrent neural networks,” CoRR, vol. abs/1308.0850, 2013. [Online]. Available: http://arxiv.org/ abs/1308.0850

    2013 arXiv

  12. [19]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” CoRR, vol. abs/1707.06347, 2017. [Online]. Available: http://arxiv.org/abs/ 1707.06347

    2017 arXiv

  13. [20]

    Openai baselines,

    P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y . Wu, and P. Zhokhov, “Openai baselines,” https://github.com/openai/baselines, 2017

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.