{"id":"9605bf91-7598-44ee-85d4-bb2c53d09bcc","arxiv_id":"1908.03936","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using the mean and covariance of similar source skills to initialize a policy search distribution makes simulated robot skill learning faster and often more accurate than random initialization.","lead":"This paper combines probabilistic movement primitives with a policy search algorithm so a simulated robot can reuse previously learned skills when facing a new pushing task. It reports that starting from similar known skills speeds up learning and cuts the number of iterations needed by more than 60 percent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 60% full-transfer speed-up claim rests on an unreported hand-tuned covariance scale s, and the paper's own Section IV-B4 shows that increasing s does not fix the k=1 failure, so the benefit may be an artifact of tuning.","rationale":"The paper's central claim is an empirical comparison: transferring past experience should improve final skill quality and reduce required iterations by more than 60% relative to a baseline. For that claim to hold, the improvement must be attributable to the transfer mechanism, not to favorable hyperparameter choices. The reader's weakest assumption identifies the hand-tuned scaling factor s, and the manuscript itself flags the relevant limitation in Section IV-B4: increasing s did not resolve the k=1 failure on dataset B, and the shape of the search space matters. That admission makes the concern concrete rather than speculative. The full-transfer speed-up is the numerical core of the abstract's 60% claim, so this is load-bearing. The paper deserves credit for an honest discussion of negative transfer and for clearly distinguishing the robust partial-transfer behavior from the more fragile full-transfer behavior, which is why the appropriate verdict remains CONDITIONAL rather than REJECT. The proposed sweep over s would settle whether the reported speed-up has a robust operating range or is an artifact of tuning.","tokens_in":10828,"tokens_out":5222,"duration_ms":60989,"concrete_test":"Reproduce the full experimental protocol (both datasets; k=1, 2, 9; and the baseline) for a logarithmic grid of s spanning at least three orders of magnitude, and report convergence iterations under a fixed criterion, for example the first iteration at which mean reward reaches 95% of its value at iteration 80. If the full-transfer k=n advantage and the approximately 60% sample reduction disappear for most tested s values, or if the reported s can only be selected using knowledge of the test tasks, then the central claim is tuning-dependent rather than a robust property of the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a more-than-60% reduction in required iterations is attached to the full-transfer mode in the Conclusion ('a well initialized full transfer saves about 60% of samples'), and that mode is directly controlled by the hand-tuned covariance scaling factor s introduced in Section III-C. s is never given a value, a selection rule, or a sensitivity analysis. The paper itself reports in Section IV-B4 that increasing s did not cure the k=1 full-transfer failure on dataset B, which indicates that the interaction between s and the source covariance, rather than transfer per se, determines whether the initialization helps. Because the baseline uses the same sI covariance while full transfer uses s/max(Sigma_k)Sigma_k, a favorable choice of s could inflate the apparent speed-up. The partial-transfer mode is more robust, but it does not carry the 60% speed-up claim. Without a sweep over s or a principled way to fix it, the 60% figure is not robustly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a framework for reusing previously learned robot movement skills when learning a new task. Skills are represented as probabilistic movement primitives (ProMPs), and relative entropy policy search (REPS) is used to adapt an initial distribution. The authors propose two transfer modes: partial transfer, which copies the mean of selected source skill parameters and uses an isotropic covariance sI, and full transfer, which also copies a scaled source covariance. The methods are evaluated on a simulated 3-DOF planar arm pushing a cylinder, comparing against a random initialization. The reported results show faster convergence and higher final reward for most transfer settings, and the paper claims a reduction of more than 60% in required iterations.","tokens_in":11039,"tokens_out":7019,"duration_ms":64495,"significance":"The paper addresses a relevant problem in robot learning: reusing skill libraries to speed up policy search. The combination of ProMPs with REPS is natural and the qualitative analysis of trajectory similarity (Figure 5) is informative. The authors honestly report the failure of full transfer with k=1 on dataset B (Section IV-B4), which is a strength of the exposition. If the empirical claims are made robust, the framework could be a practical component for lifelong robot learning. However, the current support for the headline speed-up claim is insufficient.","major_comments":[{"comment":"The headline claim of a reduction of more than 60% in required iterations rests on the full-transfer mode, whose covariance is scaled by the hand-tuned factor s as s/max(Σk)Σk. The paper never reports the value of s, provides no selection rule, and gives no sensitivity analysis. Moreover, Section IV-B4 states that increasing s did not resolve the k=1 full-transfer failure on dataset B, indicating that the interaction between s and the source covariance, rather than transfer per se, determines the outcome. A sweep over s (or a principled way to fix it) is necessary to establish that the speed-up is a property of the method and not an artifact of the chosen scaling.","section":"Section III-C and Section IV-B4"},{"comment":"The claim of 'more than 60%' sample reduction is not precisely quantified and appears inconsistent with the reported numbers. The text states that full transfer converges within 'latest 25 iterations' while the baseline converges around iteration 60, which corresponds to roughly 58% reduction, not more than 60%. No definition of 'required iterations' (for example, the iteration at which reward reaches a threshold or a stopping criterion) is given, and no statistical test or confidence interval is provided for the comparison. The abstract's unqualified statement that 'required iterations ... can be reduced by more than 60%' is therefore not supported by the evidence.","section":"Section IV-B1 and Conclusion"},{"comment":"The REPS KL constraint epsilon is a free parameter that directly controls the step size and thus the convergence speed; its value is never reported. Similarly, the reward weights a and b are hand-tuned (Section IV-A3) and their values are not given. Since the evaluation compares reward curves across settings, the omission of these parameters makes the experiments irreproducible and prevents assessing whether the observed speed-ups are specific to the chosen hyperparameters. Please report all hyperparameters and, where feasible, provide a sensitivity analysis.","section":"Section III-B and Section IV-A3"},{"comment":"The abstract claims that 'the quality of the learned skills improves and the required iterations ... can be reduced by more than 60% when past experiences are utilized.' Yet the paper's own results show that the full-transfer mode with k=1 on dataset B fails to outperform the baseline, and the partial-transfer mode does not carry the 60% speed-up. The conclusion restricts the 60% figure to 'a well initialized full transfer,' but the abstract is unqualified. The empirical claims need to be scoped to the settings in which they hold.","section":"Abstract and Section IV-B4"}],"minor_comments":[{"comment":"The description of the reward weights says a and b are tuned so that the mean of rT is 1.5 times the mean of rp. Please report the actual values and the procedure used to estimate these means, as this affects the reward scale in Figure 2.","section":"Section IV-A3"},{"comment":"The baseline initialization is defined as µb = λµN + (1−λ)µr with λ based on the basis functions at time zero. The rationale for this particular baseline should be explained more clearly, since it may already contain a form of prior knowledge through µN.","section":"Section III-C"},{"comment":"The sentence 'Each of the transfer possibilities is evaluated five times. This results in overall 5×460 = 2300 evaluations' does not explain the origin of the factor 460. Please clarify how many transfer possibilities are counted.","section":"Section IV-B"},{"comment":"The sentence 'The convergence takes approximately 60 iterations, which is about the same number of iterations as the baseline' is ambiguous because the antecedent is the partial transfer with k=1. Please rephrase to clarify.","section":"Section IV-B1"},{"comment":"The lower row would benefit from a caption explaining what the yellow bars and red dots represent in more detail, especially for readers not familiar with ProMP weight visualization.","section":"Figure 5"},{"comment":"The distance measure from [30] is cited to the authors' own earlier work; adding a one-sentence description of the measure would make the paper self-contained.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a robotics or machine learning venue. The main empirical claims need strengthening; the authors should be encouraged to run a sensitivity analysis and report all hyperparameters. The work is not fundamentally flawed, but the current version overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the clean empirical split between partial transfer (mean only) and full transfer (mean plus covariance) when initializing REPS from a skill library. The building blocks are all known, but the explicit comparison, and the finding that partial transfer is more robust to poor source selection while full transfer converges faster and preserves trajectory shape, is genuinely new and believable. The paper is also unusually candid: it reports the k=1 full transfer failure on dataset B, admits that increasing the scaling factor s did not fix it, and discusses negative transfer without spinning it.\n\nThe biggest soft spot is exactly what the stress-test note flags: s is hand-tuned and never given a value, a selection rule, or a sensitivity analysis. Since full transfer uses s/max(Sigma_k)Sigma_k, the claimed 60% sample savings could be partly an artifact of the chosen scale. The paper's own discussion shows that s alone does not control the failure mode, so the interaction between s and the source covariance is doing real work. That hole weakens the headline number. The reward weights a and b are also unreported, and there are no significance tests, with only five runs per setting. The baseline is not knowledge-free either: it mixes the average of all source means into the initialization, which makes it a strong baseline rather than a random one. That is arguably conservative, but it should be labeled honestly. No code or data are released, so the numbers cannot be checked.\n\nThese are addressable issues, not fatal ones. The central observation that transfer helps, especially when the library is large, is supported by the figures across two datasets, and the honest treatment of failure is more than most papers offer. A serious referee can extract useful signal here.\n\nWho is this for? People working on policy search initialization, robot skill libraries, or transfer in RL. It is a solid empirical contribution, not a methodological breakthrough. I would send it to review, with the expectation that the authors report s, a, b, run a small sweep over s, and ideally release code or data. The qualitative conclusions will likely hold; they just need to be pinned down.","headline":"A usefully honest empirical comparison of mean-only versus full-covariance ProMP initialization for REPS, whose headline 60% speed-up claim is held hostage by an unreported hand-tuned scaling parameter.","tokens_in":11501,"tokens_out":2684,"would_cite":true,"duration_ms":32718,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Initializing policy search with similar movement primitives reduces required learning iterations by more than 60% and improves final skill quality.","keywords":["transfer learning","probabilistic movement primitives","relative entropy policy search","policy initialization","robot skill learning","k-nearest neighbors","lifelong learning","object pushing"],"falsifier":"Sweep the scaling factor s over a wide range for the full-transfer mode with k=1 on the harder starting-pose dataset and count iterations to a fixed reward threshold. If no value of s reaches the random baseline's final reward, the claimed more-than-60% iteration saving fails for exactly the setting the paper itself flags as problematic.","tokens_in":10651,"feed_emoji":"🤖","tokens_out":7268,"duration_ms":76605,"temperature":0.7,"pith_summary":"This paper argues that a robot learning a new motor skill should start from the solutions of similar skills already in its library rather than from scratch. It combines probabilistic movement primitives, which represent a skill as a Gaussian over trajectory parameters, with a description of each skill's effect, so the most relevant past skills can be selected by comparing task descriptors. The selected skill parameters initialize the search distribution of relative entropy policy search, and the paper reports that this initialization improves the final quality of the learned pushing skill and reduces the required learning iterations by more than 60% on a simulated 3-DOF planar arm. Two transfer modes are compared: transferring only the source mean, which is robust and reaches a higher final reward, and transferring the full source covariance, which converges fastest and keeps movements similar to known safe ones but depends on having several good source skills.","feed_headline":"Reusing past robot skills cuts learning time by 60%","feed_subtitle":"Initializing policy search with similar movement primitives improves final skill quality and speeds convergence.","key_machinery":"The load-bearing objects are Probabilistic Movement Primitives (ProMPs), a skill stored as a Gaussian over the weights of shared basis functions, giving both a mean trajectory and a trajectory covariance, and Relative Entropy Policy Search (REPS), a sample-based optimizer that repeatedly fits a Gaussian to reward-weighted samples while limiting the KL divergence from its previous distribution. The framework adds a task descriptor to each ProMP and uses Euclidean distance between descriptors with k-nearest neighbors to select source skills. The transfer step itself is the mechanism: the source mean (partial transfer) or the source mean plus scaled covariance (full transfer) is inserted as REPS's initial search distribution, and a hand-tuned scaling factor s keeps that distribution from being too restrictive.","core_discovery":"On the paper's own terms, the central discovery is that the initial distribution of a policy search matters as much as the search algorithm itself, and that a probabilistic skill library can supply that initialization. New skills are learned by taking the mean (partial transfer) or the mean and scaled covariance (full transfer) of the k nearest source skills, chosen by Euclidean distance between task descriptors, and using that as the start of episode-based REPS. Across ten simulated object-pushing tasks, both modes beat a random initialization on final reward in most settings, and full transfer converges within about 25 iterations, saving roughly 60% of samples compared with the baseline's approximately 60 iterations. Full transfer also preserves the qualitative shape of the source trajectories, which the authors identify with safer, more predictable exploration; however, with a single forced nearest neighbor it can get trapped in a local optimum and underperform the baseline, so the benefit is conditional on source quality and library size.","pith_inferences":["The paper does not test whether the iteration saving transfers to other policy-search algorithms; a natural check would be to replace REPS with any KL-regularized distribution-based optimizer and see whether the saving persists.","The k=1 full-transfer failure suggests a safeguard the paper does not implement: monitor the KL divergence between the source distribution and the REPS updates, and switch to partial transfer when the search distribution stays pinned to the source.","For a lifelong-learning curriculum, the result implies an ordering rule: learn a cluster of similar tasks before a distant one, because full transfer is reliable only when the library contains several comparable source skills."],"forward_implications":["Full transfer with a well-stocked library can reach the final reward in about 25 iterations instead of the baseline's roughly 60, saving about 60% of the required samples.","Learning from the mean of the closest known skills (partial transfer) reliably outperforms random initialization in final reward regardless of library size, making it the safer default when similar source skills are scarce.","Full transfer preserves the shape of the known movement, so exploration stays near previously safe trajectories, at the cost of sometimes settling for a lower reward than partial transfer reaches.","Transferring the full covariance only pays off when several similar source skills are available; with a single forced source it can get stuck in a local optimum and fall below the random baseline.","Because REPS treats the reward as a black box, the framework needs no model of the object–arm contact dynamics to benefit from the initialization."],"supporting_citations":[{"why":"Supplies the ProMP representation of a skill as a distribution over trajectory parameters.","marker":"[2]"},{"why":"Gives the probabilistic movement primitive formulation that the skill library stores and transfers.","marker":"[19]"},{"why":"Provides the REPS optimizer whose search distribution the framework initializes with source knowledge.","marker":"[22]"},{"why":"Provides the episode-based REPS formulation and policy-search background the experiments build on.","marker":"[23]"},{"why":"Defines the three transfer benefits—higher start, higher asymptote, higher slope—used to evaluate the results.","marker":"[16]"},{"why":"Supplies the k-nearest-neighbor rule that selects similar source skills by task-descriptor distance.","marker":"[27]"},{"why":"Provides the physics simulation in which the planar pushing tasks and rewards are evaluated.","marker":"[29]"},{"why":"Supports using Euclidean distance between task descriptors as the similarity measure for source selection.","marker":"[30]"}],"fun_headline_variants":["Robot skill reuse cuts learning time by 60%","Initialize policy search with past skills for 60% faster learning","Past skills as priors speed robot learning by 60%","Knowledge transfer in movement primitives saves 60% of samples","Reusing similar skills improves final policy and cuts iterations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported savings rest on a single manually tuned number s that controls how wide the initial search is; if that number must be retuned for each task or library, the full-transfer speed-up is not a robust property of the method.","fun_headline_variants_meta":{"raw":{"variants":["Robot skill reuse cuts learning time by 60%","Initialize policy search with past skills for 60% faster learning","Past skills as priors speed robot learning by 60%","Knowledge transfer in movement primitives saves 60% of samples","Reusing similar skills improves final policy and cuts iterations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1206,"prompt_tokens":903,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":519,"tokens_out":303,"duration_ms":4011,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:57:35.669371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sweep the scaling factor s over a wide range for the full-transfer mode with k=1 on the harder starting-pose dataset and count iterations to a fixed reward threshold. If no value of s reaches the random baseline's final reward, the claimed more-than-60% iteration saving fails for exactly the setting the paper itself flags as problematic.","supporting_citations":[{"cited_title":"Probabilistic movement primitives,","cited_arxiv_id":null,"evidence_quote":"Supplies the ProMP representation of a skill as a distribution over trajectory parameters."},{"cited_title":"Using proba- bilistic movement primitives in robotics,","cited_arxiv_id":null,"evidence_quote":"Gives the probabilistic movement primitive formulation that the skill library stores and transfers."},{"cited_title":"Relative entropy policy search,","cited_arxiv_id":null,"evidence_quote":"Provides the REPS optimizer whose search distribution the framework initializes with source knowledge."},{"cited_title":"A survey on policy search for robotics,","cited_arxiv_id":null,"evidence_quote":"Provides the episode-based REPS formulation and policy-search background the experiments build on."},{"cited_title":"Transfer learning,","cited_arxiv_id":null,"evidence_quote":"Defines the three transfer benefits—higher start, higher asymptote, higher slope—used to evaluate the results."},{"cited_title":"An introduction to kernel and nearest-neighbor nonparametric regression,","cited_arxiv_id":null,"evidence_quote":"Supplies the k-nearest-neighbor rule that selects similar source skills by task-descriptor distance."},{"cited_title":"Pybullet, a python module for physics simulation for games, robotics and machine learning","cited_arxiv_id":null,"evidence_quote":"Provides the physics simulation in which the planar pushing tasks and rewards are evaluated."},{"cited_title":"A comparison of distance measures for learning nonparametric motor skill libraries,","cited_arxiv_id":null,"evidence_quote":"Supports using Euclidean distance between task descriptors as the similarity measure for source selection."}],"review_version":1}