{"id":"2a2e6e25-4ce9-4404-adc4-4307205c669f","arxiv_id":"1908.03731","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A learned generative model of past task motions, used as exploration noise in DDPG, speeds up learning of new robot manipulation and contact tasks by more than two times in simulation.","lead":"The authors train a neural network on motions from previously solved robot tasks and use its outputs to guide exploration while learning new tasks, instead of random noise. In simulated robot arm experiments, this cut learning time by more than half on several tasks, especially when rewards were sparse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The speedup claim rests on an untested additivity assumption (Remark 1): LEP is trained to emit full actions but deployed as additive noise pi(s)+epsilon_LEP; if the sum leaves the training-action distribution, the learned prior is not what drives exploration.","rationale":"The reader's weakest_assumption identifies the transferability of the LEP and, as a second premise, the Remark 1 additivity issue. I think the additivity premise is the more load-bearing one: the point-mass argument in Section II-A and the likelihood objective justify LEP as a generator of full actions, but Eq. (5) uses it as an additive perturbation, which is an extra structural assumption. The paper explicitly disclaims a guarantee in Remark 1, making this the place where the central speedup claim is least secure. The evaluation weaknesses (best-configuration selection, no significance tests, no code) are real but secondary; the robustness experiment in Section IV-D mitigates some selection bias. The concern does not overturn the verdict: the simulations are consistent and the method is plausible, but the mechanism should be demonstrated before the speedup is taken at face value. Therefore I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":6,"tokens_out":6342,"duration_ms":196210,"concrete_test":"Re-run Tasks 2 and 4 with fixed seeds, logging every executed command u_t=pi(s_t)+epsilon_LEP. Compute (a) the fraction of u_t outside the 5th-95th percentile envelope of actions in the LEP training set, (b) the fraction saturating the torque limits, and (c) the speedup obtained when epsilon_LEP is scaled by 0.1. Include a matched baseline whose Gaussian/OU noise has the same marginal variance as epsilon_LEP. If a large fraction of executed commands lies outside the training envelope, or if the 0.1-scaled version retains the speedup, the additive assumption is the load-bearing component and the learned prior is not responsible for the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-A trains LEP to model p(a|s_{1:t}) over full actions from previous policies. Section II-B (Eq. 5) then executes u_t = pi(s_t)+epsilon_LEP, i.e. a full LEP action is added to the current policy output. For the transfer claim to hold, u_t must stay in the part of action space represented by the LEP training data. Remark 1 concedes 'there is no guarantee that the sum with output of the current policy will do the same,' and the only justification is that the policy is initialized near zero. As training progresses, pi changes; in sparse-reward contact tasks (Tasks 2 and 4) the composite command can saturate torque limits or produce contact forces outside anything in the training set. Then exploration is no longer driven by learned motion primitives, and the reported speedup could stem from raw noise amplitude or from the policy itself rather than from the transfer mechanism. No distributional analysis or ablation distinguishes these cases.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a learned exploration process (LEP) for deep reinforcement learning on robotic manipulation and contact tasks. A recurrent network is trained on short trajectories from previously solved tasks to model action sequences conditioned on state histories, and the resulting samples are added to the deterministic policy of DDPG in place of standard exploration noise (Section II-B, Eq. (5)). The authors report experiments on four simulated KUKA LWR tasks: reaching, stationary force application, periodic motion, and periodic motion with contact force regulation. They claim that the approach more than doubles learning speed relative to standard DDPG and PPO when rewards are sparse, while remaining robust to parameter choices and enabling continuous learning as new tasks are solved.","tokens_in":9957,"tokens_out":3351,"duration_ms":37832,"significance":"If the central claim holds, the paper offers a practical way to reuse trajectory data from prior tasks as an exploration prior without modifying reward functions or task structure. The idea is well motivated by the point-mass example, and the experimental design covers a meaningful progression from free-space reaching to contact-rich periodic motion with sparse rewards. The paper also reports a helpful negative result: on a dense-reward periodic task whose motion profile differs from the training data (Task 3, Figure 5), LEP does not speed up learning but also does not hurt performance. The four-task evaluation, the continuous-learning update of the model, and the robustness check in Figure 7 are strengths. However, the manuscript does not ship code, hyperparameter tables, or seed-level statistics, so the experiments are not fully reproducible from the text alone, and the causal interpretation of the speedup is not yet fully supported.","major_comments":[{"comment":"The mechanism behind the reported speedups is asserted rather than demonstrated. The LEP is trained to reproduce full action sequences (Section II-A), yet at deployment it is added to the current deterministic policy as u_t = pi(s_t) + epsilon_LEP. Remark 1 concedes that \"there is no guarantee that the sum with output of the current policy will do the same,\" and the only supporting argument is that the policy is initialized near zero. Because the policy changes during training and the tasks involve torque limits and contact forces (Tasks 2 and 4), the composite command can leave the distribution of actions on which the LEP was trained. Without either a distributional analysis of u_t relative to the LEP training data, an ablation that compares LEP against noise of matched amplitude or variance, or a variant that learns residual noise conditioned on pi(s_t), the paper does not establish that the learned motion statistics, rather than the altered noise structure, cause the speedups.","section":"II-B, Eq. (5), Remark 1"},{"comment":"The quantitative speedup claim rests on an evaluation protocol that is not fully specified. The learning curves in Figures 3-6 show the best configuration per algorithm, while Section III-C defines the success threshold empirically from observed behaviors. No significance tests or confidence intervals are reported, and the number of random seeds is not stated despite the paper citing reference [21] on seed variability. Figure 7 shows robustness to parameterization only for Task 4 and not for the other tasks. To support the \"more than double learning speed\" claim, the authors should report aggregate results across all tested configurations and seeds, state the success-threshold selection rule, and test sensitivity of the measured speedups to the chosen threshold.","section":"III-C, IV-A, IV-D"},{"comment":"Section II-C describes a continuous learning loop in which the exploration model is repeatedly expanded as new tasks are solved, but Section IV-C tests only a single additional round of training: one LEP model is trained on data from Tasks 2 and 3 and then evaluated on Task 4. The abstract's claim that the approach \"enables continuous learning of improved exploration strategies as novel tasks are learned\" is therefore not supported by an experiment with multiple expansions. An additional experiment with two or more successive updates of the exploration model would be needed to substantiate this claim.","section":"II-C, IV-C"}],"minor_comments":[{"comment":"The notation s_{1:t} is used in the point-mass example to denote both full trajectories and shorter windows, while the training procedure uses segments of length h. Please clarify the conditioning variable precisely and specify whether the LEP is conditioned on states only or on state-action pairs.","section":"II-A"},{"comment":"The gray line labeled \"Task success\" and the vertical dashed lines marking convergence are not defined in the captions. Please state how the success threshold is determined from the reward and how the convergence point is computed from the learning curves.","section":"Figures 3-6"},{"comment":"The empirical success threshold is described only qualitatively. Please provide a table of the threshold values for all four tasks and justify that the reported speedups are not sensitive to small changes in these thresholds.","section":"III-C, IV-A"},{"comment":"The number of random seeds used for each learning curve is not reported, and it is unclear whether the \"100 policies\" used to train LEP are independent training runs or multiple snapshots from a few runs. Please report the number of seeds and the variance across seeds.","section":"IV-A, IV-C"},{"comment":"No hyperparameter table or code release is provided, which makes the experiments difficult to reproduce. Please include a table with network architectures, learning rates, replay buffer sizes, action noise parameters, and the LEP subsequence lengths tested.","section":"Experimental Setup"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a robotics or machine learning venue and the core idea is interesting. The main concerns are the unvalidated additivity assumption in Eq. (5) and the evaluation protocol (best-configuration selection, empirically defined success thresholds, no seed reporting). Both are addressable with additional experiments and analysis, so I recommend major revision rather than rejection. The authors cite reference [21] on seed variability but do not follow its recommendations; this should be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This paper does something actually new: it trains an LSTM generative model on short state-action segments from previously solved tasks and uses samples from it as the exploration noise in DDPG. The point-mass argument for conditioning on history rather than a single state is a useful conceptual step, and it explains the design in a way most such papers don't. Second, the empirical claims are likely a bit too strong as stated, but the core mechanism probably works.\n\nWhat's good: the writing is honest. Remark 1 explicitly concedes that there is no guarantee the sum of the deterministic policy output and the LEP sample stays in the training distribution. The experiments cover a sensible range—reaching, force application, periodic motion, and periodic motion with contact—and the speedups on sparse-reward tasks are consistent across task instances. The robustness plot (Fig. 7) is a real plus: it shows the method is less sensitive to hyperparameters than DDPG, which is a useful practical claim independent of the speedup.\n\nWhere it's soft: the evaluation protocol. Best configuration per algorithm is chosen, success thresholds are defined post hoc, and there are no significance tests. That weakens the \"more than twice as fast\" claim. No code or data is released, so reproducibility is limited. The stress-test concern about additivity is legitimate. LEP is trained to produce full actions, but deployed as pi(s) + epsilon. Early in training, policy output is near zero so the samples are pure LEP; later, the sum can drift out of the learned distribution. The paper acknowledges this but offers no analysis. An ablation comparing LEP noise to random noise of matched amplitude, or a version trained to output residuals, would settle whether the transfer mechanism is doing the work or just high-variance noise. Task 3 helps here: when the required motion profile is very different, LEP gives no speedup, which suggests the mechanism is not just raw noise.\n\nOverall, the central idea is solid, the paper is clearly written, and the limitations are mostly disclosed. The gaps are in evaluation rigor, not in the conceptual contribution. The right reader is someone working on transfer learning or exploration in RL; they'll find the history-conditioning argument worth engaging with. My recommendation: send it to peer review, with major revision focused on significance testing and at least one ablation that isolates the additive-noise issue.","headline":"A genuinely new exploration mechanism with honest empirical support, but the additive-noise assumption is untested and the evaluation protocol is too loose for the speedup claims as stated.","tokens_in":10428,"tokens_out":1912,"would_cite":true,"duration_ms":22296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing random exploration noise in DDPG with samples from a generative model trained on previously solved tasks more than doubles learning speed on new motion and contact tasks, especially when rewards are sparse.","keywords":["learned exploration","generative model","DDPG","LSTM","sparse rewards","robot manipulation","contact tasks","transfer learning"],"falsifier":"Train LEP only on reaching trajectories and measure episodes-to-success on a sparse-reward contact task over many random seeds; if LEP-equipped DDPG does not beat tuned DDPG with Ornstein-Uhlenbeck noise, the transfer claim is not supported. To isolate the mechanism, repeat with LEP trained on random or non-informative trajectories and check that the speedup disappears, which would show the effect comes from task structure rather than from the recurrent network itself.","tokens_in":9539,"feed_emoji":"🤖","tokens_out":7190,"duration_ms":74759,"temperature":0.7,"pith_summary":"The paper tries to show that a robot can learn new tasks faster by reusing the motions it already knows, without reshaping the reward. It replaces the random exploration noise in a standard deep reinforcement learning algorithm for continuous control with samples from an LSTM that generates short action sequences conditioned on recent state history, trained on trajectories from previously solved tasks. In simulated 7-degree-of-freedom manipulator tasks, from reaching to periodic motion with contact force regulation, this learned exploration process more than doubles learning speed compared with standard DDPG using Gaussian or Ornstein-Uhlenbeck noise, and the gain is largest when rewards are sparse. The authors also claim that the method is robust to parameter tuning and can continually update the exploration model as new tasks are solved.","feed_headline":"Reusing old robot motions doubles learning speed","feed_subtitle":"A generative model of past behaviors replaces random exploration noise, with the biggest gains on sparse-reward contact tasks.","key_machinery":"The Learned Exploration Process (LEP) is a recurrent LSTM network trained by maximum likelihood on short, randomly sampled $h$-step sub-trajectories taken from policies that solved previous tasks. It maps a history of states to the parameters of a diagonal Gaussian distribution over the corresponding action sequence. This machinery carries the argument: conditioning on state histories instead of a single state keeps the combined distribution of previous policies focused, avoiding the uniform and uninformative action distribution that arises when only the current state is used. In the reinforcement learning loop, LEP replaces the independent noise process in DDPG, so each action is the current deterministic policy plus one LEP sample, with the LSTM internal state reset every $h$ steps to match the training sequence length.","core_discovery":"The central discovery is that exploration noise can itself be learned from past experience instead of being sampled from a generic, unstructured process. Conditioning a combined policy distribution on a short history of past states, rather than on the current state alone, concentrates the distribution into behaviors resembling those of the previously learned policies; the paper illustrates this with a point-mass reach task and then instantiates it as the Learned Exploration Process (LEP). In DDPG, where actions are computed as $a_t = \\pi(s_t) + \\epsilon$, LEP replaces the external noise source $\\epsilon$ with a sample from an LSTM-predicted diagonal Gaussian over action sequences, reset every $h$ steps. Across four simulated tasks on a 7-DoF arm, the LEP-equipped DDPG converges at least twice as fast on the sparse-reward contact tasks and produces more consistent final policies, while on a periodic-motion task whose movement profiles differ from the training data it performs no worse than standard DDPG.","pith_inferences":["A testable extension is to vary the history length $h$: the point-mass argument predicts that as $h$ shrinks toward zero the learned exploration distribution should flatten toward the uninformative uniform distribution, so learning curves across $h$ would directly test the proposed mechanism.","If the speedup stems from injecting temporally correlated, low-level motion structure, the same principle could transfer to other off-policy algorithms such as soft actor-critic, or to sim-to-real settings where safe motion priors constrain exploration.","Since LEP needs only trajectories, a natural next step is to assemble the exploration prior from offline datasets of previously demonstrated or optimized behaviors, potentially removing the need for any prior reinforcement learning on the system.","The robustness results suggest that replacing unstructured noise with structured samples reduces variance across random seeds; an experiment comparing seed variance of LEP-DDPG against standard DDPG across many tasks would quantify this effect directly."],"forward_implications":["New tasks can be learned more than twice as fast when rewards are sparse, provided the required movement profiles resemble those present in the training data.","The exploration model can be retrained with data from newly learned tasks, so a curriculum of tasks from simple to complex can progressively enrich the exploration strategy.","Because only trajectories are needed to train LEP, the exploration prior can be built from any source of good behavior, including trajectory optimization or demonstration, not just from full policies.","The method reduces sensitivity to parameter tuning and random initialization, making final performance more consistent across task instances and hyperparameter settings.","The approach is not tied to DDPG: any off-policy algorithm with independent exploration noise could in principle adopt the same learned exploration process."],"supporting_citations":[{"why":"Defines the DDPG algorithm whose exploration noise is replaced by LEP samples and serves as the primary baseline.","marker":"[4]"},{"why":"Provides the LSTM architecture used to model action sequences conditioned on state histories.","marker":"[14]"},{"why":"Supplies the maximum-likelihood training objective for the recurrent generative model of action sequences.","marker":"[18]"},{"why":"Defines PPO, the on-policy algorithm used as a comparison baseline.","marker":"[19]"},{"why":"Supplies the baseline implementations used for DDPG and PPO in all experiments.","marker":"[20]"},{"why":"Defines the Ornstein-Uhlenbeck correlated noise process that is the main alternative exploration strategy compared against LEP.","marker":"[3]"},{"why":"Documents how random seeds affect deep reinforcement learning results, motivating the paper's robustness evaluation across initial conditions and parameter settings.","marker":"[21]"}],"fun_headline_variants":["Learned noise speeds robot learning 2×","Past moves teach robots to explore better","Exploration learned from old tasks halves training time","Robots learn to explore by recalling past moves","Sparse rewards? Learned exploration doubles speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that short segments of behavior from previously solved tasks contain motion regularities that transfer to new tasks on the same robot, with a brief state history enough to expose them, and that adding these samples to the changing deterministic policy remains sensible throughout training.","fun_headline_variants_meta":{"raw":{"variants":["Learned noise speeds robot learning 2×","Past moves teach robots to explore better","Exploration learned from old tasks halves training time","Robots learn to explore by recalling past moves","Sparse rewards? Learned exploration doubles speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2139,"prompt_tokens":873,"completion_tokens":1266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1198}},"tokens_in":489,"tokens_out":1266,"duration_ms":17266,"temperature":1.0,"reasoning_tokens":1198,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:03:22.622565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LEP only on reaching trajectories and measure episodes-to-success on a sparse-reward contact task over many random seeds; if LEP-equipped DDPG does not beat tuned DDPG with Ornstein-Uhlenbeck noise, the transfer claim is not supported. To isolate the mechanism, repeat with LEP trained on random or non-informative trajectories and check that the speedup disappears, which would show the effect comes from task structure rather than from the recurrent network itself.","supporting_citations":[{"cited_title":"Openai baselines,","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline implementations used for DDPG and PPO in all experiments."}],"review_version":1}