Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Simulation-Aided Policy Tuning for Black-Box Robot Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By keeping a posterior over the objective's gradient and querying until a policy update is statistically guaranteed to improve, this paper makes simulator-assisted black-box policy search work on a real robot with fewer than half the…

desk verdict Useful extension of GIBO with an adaptive stop rule and a simulator channel, but the central improvement guarantee is not instantiated in experiments because the Lipschitz constant L is never set or estimated. read the letter →

arxiv 2411.14246 v1 pith:XIG5LP5D submitted 2024-11-21 cs.RO cs.LGcs.SYeess.SY

Shiming He , Alexander von Rohr , Dominik Baumann , Ji Xiang , Sebastian Trimpe This is my paper · ORCID
classification cs.ROcs.LGcs.SYeess.SY
keywords Bayesianoptimizationblack-boxpolicysearchsim-to-realtransfermulti-fidelityGaussianprocessesmovementprimitivesrobotlearningderivative-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to make black-box policy search—tuning a robot controller by observing only the reward of trial runs—practical for real hardware, where every evaluation is expensive. Its proposal, HCI-GIBO, keeps collecting data until the posterior belief over the objective's gradient makes it highly probable that the next policy update will improve performance, instead of fixing the number of evaluations per update in advance. S-HCI-GIBO adds a second, cheaper information source: an imperfect simulator that is queried first and only abandoned when its data stops shrinking gradient uncertainty. The authors demonstrate on a real robot manipulator balancing a pendulum that the simulator-aided version reaches 80 percent of the maximal reward with less than half the real-robot queries of the simulator-free version. The significance would be that fast adaptation on physical robots, previously limited to low-dimensional or heavily engineered settings, becomes feasible for higher-dimensional movement-primitive policies.

What carries the argument

The carrying object is the derivative Gaussian process, a joint GP over zeroth-order observations and the gradient at the current best policy $\hat\theta^*$; conditioning it on the data gives a closed-form Gaussian posterior on the gradient, so the gradient-information acquisition function can score candidate query points by how much they shrink the trace of the gradient covariance without knowing the outcome. The descent rule uses the posterior mean as the step direction, and the stopping rule uses Lemma 1: a step of size $\eta$ in the normalized direction $\mu/\|\mu\|$ improves the cost whenever the true gradient's projection onto that direction exceeds $\tfrac{L}{2}\eta\|\mu\|$, where $L$ bounds the gradient's Lipschitz constant. Theorem 1 replaces the true gradient by the Gaussian belief $\nabla F(\hat\theta^*) \sim \mathcal{N}(\mu,\Sigma)$, so the probability in condition (19) is a tail probability of a univariate normal, making the commitment test computable. The simulator extension is carried by a dual-source kernel $K(\bar X, x^*) = K_f(\theta,\hat\theta^*) + k_\delta(\mathrm{IS}, \mathrm{IS}^*)K_m(\theta,\hat\theta^*)$, which couples robot and simulator observations, plus a switching rule that moves from simulator to robot when consecutive simulator queries reduce the robot-gradient uncertainty by less than the threshold $\beta$.

What would settle it

Run the pendulum tracking task of Section VIII-B with $\alpha=0.95$ and $\eta=0.2$ while setting $L$ too small (e.g., by a factor of ten); if the measured frequency of reward-decreasing updates exceeds $1-\alpha=0.05$, the guarantee fails in the regime the paper leaves unspecified. Even without a new run, counting per-update successes in the reported data would settle it: the paper plots average reward but does not report the fraction of updates that actually decreased reward.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that a policy-update rule can be made both data-efficient and trustworthy: query the objective only as long as needed for the probability that the gradient step improves the function to reach a user-set confidence alpha, and then step. Theorem 1 turns Lemma 1's deterministic descent condition into a probabilistic one by replacing the true gradient with the Gaussian posterior over the gradient, so the commitment condition is computable in closed form. The S-HCI-GIBO extension says that before spending real-robot queries, the optimizer should exhaust the gradient information available from a biased simulator, using a dual-source derivative GP whose kernel couples the two sources, and switch to the robot only when another simulator query would reduce the robot-gradient uncertainty by less than a threshold beta. The real-robot pendulum experiment is then presented as evidence that this scheme learns an impromptu Lissajous tracking task from a zero-initialized dynamic movement primitive, and that the simulator-aided run reaches 80 percent of the maximum reward with less than half the real-robot evaluations of HCI-GIBO alone (58 vs 132 evaluations in Table I).

Load-bearing premise

The guarantee that each update improves with probability at least $\alpha$ requires the gradient Lipschitz constant $L$ from Assumption 1 to be known and set correctly, yet the experimental sections report $\alpha$ and $\eta$ but never $L$, so the stated confidence is not instantiated in the experiments.

Editorial extensions

If this is right

  • With the reported confidence level $\alpha=0.95$ on the pendulum hardware, each policy update should decrease reward with probability at most 0.05, so the per-update failure rate is a checkable prediction of the method.
  • Simulator data is not just a warm start: on the 24-dimensional pendulum task, reaching 80 percent of the maximal reward takes 58 real-robot evaluations with S-HCI-GIBO versus 132 without, a reduction by more than half.
  • The commitment rule replaces the fixed per-update query count of GIBO, so the number of evaluations per update adapts to the local landscape; the synthetic benchmarks show the advantage growing in 20- to 52-dimensional domains.
  • The method is a local fine-tuner, not a global planner: it assumes a reasonable initial policy (from a movement primitive or pre-trained RL) and improves it, so its success on hardware does not transfer to cold-start global discovery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same query-until-confident gradient rule could be applied to other expensive feedback channels, such as human preference ratings, where each query is a person's judgment; the only ingredient needed is a GP prior on the objective and its gradient.
  • A testable extension would be to estimate $L$ online from the GP's posterior and shrink the step size $\eta$ when the local surface is flat; the paper fixes $\eta$ and leaves $L$ unset, so confidence $\alpha$ and actual improvement frequency can drift apart.
  • The dual-source kernel's switching rule assumes one static simulator; extending the same formula to multiple simulators of different fidelity, or to a simulator whose bias drifts over time, would follow naturally but is not analyzed here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes HCI-GIBO, a local black-box policy-search algorithm based on Gradient Information with Bayesian Optimization (GIBO), and S-HCI-GIBO, a multi-fidelity extension that uses a simulator as an additional information source. The key idea is to replace GIBO's fixed number of inner-loop queries with a commitment rule derived from a Gaussian-process posterior over the objective's gradient: the algorithm queries until the probability that a gradient step improves the objective exceeds a threshold alpha (Theorem 1, Eq. (19)). S-HCI-GIBO models robot and simulator returns with a dual-source GP kernel and switches from simulator to real-robot queries when the marginal gradient information of a simulator query falls below beta. The paper reports synthetic within-model benchmarks up to 52 dimensions, fine-tuning experiments on three 'fancy gym' environments, and a real Franka Emika robot experiment for trajectory tracking of a pendulum, where S-HCI-GIBO is claimed to reach 80% of maximal reward with fewer than half the robot queries of HCI-GIBO.

Significance. The proposed commitment rule is a principled way to adapt the number of queries per gradient step and could improve data efficiency in black-box policy search; the multi-fidelity formulation with a binary information-source indicator is a natural extension of GIBO. The paper is strengthened by open-source code, a released ROS package, and a real-hardware evaluation with an imperfect simulator. However, the theoretical guarantee is conditional on the Lipschitz constant L and on the GP model being well specified, and the experimental sections do not instantiate L; there is also an inconsistency in the definition of the reality gap and the resulting kernel. These issues do not invalidate the algorithmic idea but must be fixed before the central claims can be accepted as stated.

major comments (3)
  1. [Section V-B, Eq. (19); Sections VII-A and VIII-B-b] The COMMITMENT criterion depends on the Lipschitz constant L, but L is never specified or estimated in any experiment. Theorem 1's guarantee therefore is not instantiated by the reported results; the claim that the algorithm 'queries until it can guarantee that the update step will improve the function with high probability' (Section V-B) is not verified. The authors should either fix a conservative L with justification, estimate L from the GP posterior or sample-path bounds, or report a sensitivity analysis over L; without one of these, Eq. (19) is not evaluable from the paper.
  2. [Section II, Eq. (3) and Assumption 1; Section V-C-2, Eq. (21)] The model definition is inconsistent. Eq. (3) sets f_sim = f + f_gap, but the text immediately after defines f_gap = f - f_sim; these differ by a sign. Moreover, kernel (21) with k_delta(IS,IS*) = IS*IS* gives Cov(f_sim(theta), f_sim(theta')) = Kf and Cov(f(theta), f(theta')) = Kf + Km, which corresponds to f = f_sim + f_gap, not f_sim = f + f_gap. Please correct the definitions and state precisely which function is the 'common' GP and which is the source-specific discrepancy; this is essential for reproducing the dual-information-source model.
  3. [Section V-B, Remark 1 and Eqs. (19)-(20)] The treatment of normalized gradient steps is ambiguous. Eq. (19) is derived for an update theta - eta*mu, with threshold (L/2)*eta*||mu||; if the descent direction is normalized to mu/||mu||, the threshold in (19) becomes (L/2)*eta, not (L/2)*eta*||mu||. The distribution in (20) is the correct one for a normalized direction, but the text does not state which form of (19) is used in the experiments. Please specify the exact update rule and corresponding threshold used in each experimental section.
minor comments (4)
  1. [Section V-B, Lemma 1] The vector nu is called a 'descent direction', but condition (10) is a positive inner product, which corresponds to an ascent direction if the update is theta - eta*nu; please use 'update direction' or clarify the sign convention.
  2. [Section VIII-B-c and Table I] The 'less than half' claim is based on five trials; please report per-trial query counts or confidence intervals for the reward levels so the reader can assess the variability of this headline number.
  3. [Fig. 12] The caption states that shaded regions on the x-axis show standard deviation, but the plot appears to show shaded regions around the curves; the caption and the plot should be aligned.
  4. [Section VII-A] The budget statement for S-HCI-GIBO is ambiguous because simulator evaluations are excluded from the x-axis of Fig. 6; please state the total simulator budget explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the improvement-confidence rule is a Bayesian decision rule, and the reported data-efficiency results are benchmarked rather than built into the equations.

full rationale

The derivation chain is self-contained. Theorem 1's improvement probability (Eq. 19) is a conditional statement: Lemma 1 gives a deterministic sufficient condition for descent, and Eq. 19 computes the posterior probability that a gradient realization satisfies that condition. The COMMITMENT rule stops querying when that probability reaches alpha. This is standard Bayesian sequential design; the 'guarantee' is explicitly conditional on Assumption 1 (GP priors and the Lipschitz constant L). No parameter is fitted to the reported rewards, and no equation is calibrated to the data-efficiency claims in Table I. The base GIBO query (Eqs. 6-7) and derivative GP posterior (Eqs. 4-5) are imported from [11], whose authors overlap with the present paper; however, [11] is a published, code-reproduced prior result, and the present contribution (adaptive COMMITMENT) is derived independently rather than reduced to that citation. The multi-fidelity kernel (Eq. 21) and SIMTOREAL rule (Eq. 23) are new and are benchmarked against external baselines (EI-BO, ARS, CRBO, BBRL-PPO), so the sim-to-real data-efficiency claim is empirically supported rather than definitionally forced. The manuscript's own Limitations section acknowledges the Lipschitz assumptions; the absence of a reported numerical L in the experiments is a correctness/instantiation gap, not a circular reduction. The within-model GP-sample benchmark favors GP-based methods but is a standard performance comparison, not a derivation of the central claim. No step makes the 'prediction' equal to its input by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central method rests on a Bayesian model of the objective and the simulator gap. The main unmeasured inputs are the GP hyperparameters, the Lipschitz constant L, and the switching threshold beta. No new physical entities are proposed.

free parameters (5)
  • L (gradient Lipschitz constant)
    Used in Theorem 1 and the commitment condition (19), but never specified in the experimental hyperparameter lists; the guarantee is not instantiated without a concrete L.
  • SimToReal threshold beta = 5 (fancy gym), 1 (robot)
    Controls when the algorithm switches from simulator queries to real-robot queries; hand-tuned per experiment.
  • GP kernel hyperparameters (Kf, Km) = robot: nu=2, lambda=0.3; Km nu=0.5, lambda=0.35. fancy gym: lengthscale 0.1, outputscale 1
    Hand-tuned kernel lengthscales and output scales; posterior and acquisition function depend on them.
  • Step size eta = 0.2 (all experiments)
    Gradient step size affects the improvement probability in (19) and convergence speed.
  • Improvement confidence alpha = 0.9 (fancy gym), 0.95 (robot)
    Commitment threshold; user-chosen.
assumptions (4)
  • domain assumption The real objective f and the reality gap f_gap are samples from known GP priors with differentiable means and Lipschitz gradients.
    Assumption 1. This is the foundation for the derivative GP model and for the improvement probability.
  • domain assumption The simulator output is f_sim = f + f_gap with f_gap independent of f, so the dual-source kernel (21) is correctly specified.
    Section V-C. If the gap is correlated with f, the posterior used for commitment is biased.
  • domain assumption The Lipschitz constant L is known to the algorithm.
    Theorem 1 and (19) require L; the paper does not state how L is obtained in experiments.
  • standard math Standard Gaussian process conditioning and derivative GP formulas.
    Equations (4)-(5), (7), borrowed from Rasmussen and Williams plus Muller et al.
invented entities (1)
  • Reality gap GP F_gap
    purpose: Models the unknown bias between simulator and real-robot objective.
    The gap is not directly measured; it is a latent GP with a chosen kernel. Its posterior is the mechanism for trading off sim and real queries, but no external falsifiable prediction is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simulation-Aided Policy Tuning for Black-Box Robot Learning." pith.science (2026). https://pith.science/paper/XIG5LP5D

@misc{pith2026241114246,
  author       = {Pith},
  title        = {Pith review of: Simulation-Aided Policy Tuning for Black-Box Robot Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIG5LP5D}},
  note         = {Machine review of arXiv:2411.14246}
}
read the original abstract

How can robots learn and adapt to new tasks and situations with little data? Systematic exploration and simulation are crucial tools for efficient robot learning. We present a novel black-box policy search algorithm focused on data-efficient policy improvements. The algorithm learns directly on the robot and treats simulation as an additional information source to speed up the learning process. At the core of the algorithm, a probabilistic model learns the dependence of the policy parameters and the robot learning objective not only by performing experiments on the robot, but also by leveraging data from a simulator. This substantially reduces interaction time with the robot. Using this model, we can guarantee improvements with high probability for each policy update, thereby facilitating fast, goal-oriented learning. We evaluate our algorithm on simulated fine-tuning tasks and demonstrate the data-efficiency of the proposed dual-information source optimization algorithm. In a real robot learning experiment, we show fast and successful task learning on a robot manipulator with the aid of an imperfect simulator.

Figures

Figures reproduced from arXiv: 2411.14246 by the authors.

Figure 1
Figure 1. Experiment setup: A robot manipulator is balancing a planar pendulum and learning to follow the reference trajectory with the pendulum. their policy updates. Further, the use of uncorrelated random noise in the action space for exploration can lead to jerky motions [4], which are undesirable in real-world settings, since actuators may take a lot of wear and tear during the training. An alternative to address the abo… view at source ↗
Figure 2
Figure 2. Sequential black-box policy search: The search algo￾rithm determines a query in order to gain more information on the performance function. The policy is then evaluated with the parameters given by the query. In each iteration, the search algorithm decides on its current guess of the best policy. robotic system. Specifically, in the experimental sections of this paper we use the parameters to define a MP. We denote … view at source ↗
Figure 3
Figure 3. Visualization of improvement confidence regions: Improvement confidence of gradient distribution ∇Fa and ∇Fb are 97 % and 76 %, respectively. Mean vectors are de￾noted by black arrows, and the contour lines show the density of the gradient distribution. If Lη = 1 and true gradients are in regions filled with dots, gradient step using the mean vector improves the policy. function for the neighboring query steps on th… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Multi-fidelity queries: Comparison of uncertainties (blue shaded regions) after queries at same locations (0.5 and 0.7 shown with star markers) from the simulator and the robot. Use toy objective function: f: blue line and fsim: orange dash￾dotted line; Posterior mean …
Figure 5
Figure 5. Figure 5: Visualization of the S-HCI-GIBO optimization process with a 1-dimensional function. Top: The objective functions of f (blue line) and fsim (orange dash-dotted line). The red cross symbol refers to the current parameter θi . The blue and orange star markers represent th…
Figure 6
Figure 6. Figure 6: Within-model comparison: Mean of the reward for six different dimensional function domains for 100 runs. Evaluations are for f and do not include queries to fsim. HCI-GIBO and S-HCI-GIBO show greater data efficiency, especially in a high dimensional setting. This appro…
Figure 7
Figure 7. Figure 7: Visualization of three fancy gym environments: hopper jumping (15-dim. domain), reacher (25-dim. domain), and box pushing (70-dim. domain). 0 100 200 300 # of evaluations 826 950 Reward Hopper Jump (Dense) 0 100 200 300 # of evaluations 19.0 21.5 Hopper Jump (Non-Marko…
Figure 8
Figure 8. Figure 8: Results of fine-tuning control policies for three fancy gym environments: Evaluations start from five random initial conditions over 20 runs. BO approaches exhibit superior refinement in task performance compared to the black-box RL. VIII. ROBOT LEARNING EXPERIMENTS We…
Figure 9
Figure 9. Figure 9: Experiment setup: The Franka Emika robot is holding a planar pendulum, and the Vicon optical motion capture tracks the pendulum object [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Pendulum: The pendulum is a stick with a needle, and the end-effector can move in all directions in the horizontal plane. B. Learning from Scratch: Impromptu Trajectory Tracking Trajectory tracking control is a fundamental and essen￾tial aspect of robotic systems, yet…
Figure 11
Figure 11. Figure 11: Experimental results: The trajectory is improved after each policy update. The reference and the actual trajectory are marked by white-dotted line and blue line, respectively. and a Vicon optical motion capture system. The Franka Emika robot is a seven-joint open-chai…
Figure 12
Figure 12. Figure 12: Progress of HCI-GIBO and S-HCI-GIBO for the robot experiment: Averaged over five trials (solid lines). We also run the HCI-GIBO on the simulator, which is denoted by the light-blue dot (“sim only”), and then deploy the learned policy on the real robot. Transferring th…
Figure 14
Figure 14. Figure 14: Improvement: The trajectory of the pole tip after each gradient step converges towards the desired one. uncertainties of gradient estimates, quantifies the improvement confidence to adaptively control the required queries for policy updates, and leverages the informat…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 39 canonical work pages

  1. [1]

    Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collec- tion,

    S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collec- tion,” The International Journal of Robotics Research , vol. 37, no. 4-5, pp. 421–436, 2018. DOI: 10 . 1177 / 0278364917710318

    work page 2018

  2. [2]

    OpenAI et al., Solving rubik’s cube with a robot hand ,

    work page

  3. [3]

    Learning quadrupedal locomotion over challenging terrain,

    J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science Robotics , vol. 5, no. 47, eabc5986, 2020. DOI: 10.1126/scirobotics.abc5986

    doi:10.1126/scirobotics.abc5986 2020

  4. [4]

    How to train your robot with deep reinforce- ment learning: Lessons we have learned,

    J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforce- ment learning: Lessons we have learned,” The Interna- tional Journal of Robotics Research , vol. 40, no. 4-5, pp. 698–721, 2021. DOI: 10.1177/0278364920987859

    doi:10.1177/0278364920987859 2021

  5. [5]

    A survey on policy search algorithms for learning robot controllers in a handful of trials,

    K. Chatzilygeroudis, V . Vassiliades, F. Stulp, S. Calinon, and J.-B. Mouret, “A survey on policy search algorithms for learning robot controllers in a handful of trials,” IEEE Transactions on Robotics, vol. 36, no. 2, pp. 328– 347, 2019. DOI: 10.1109/TRO.2019.2958211

    arXiv 2019

  6. [6]

    Move- ment imitation with nonlinear dynamical systems in humanoid robots,

    A. J. Ijspeert, J. Nakanishi, and S. Schaal, “Move- ment imitation with nonlinear dynamical systems in humanoid robots,” in IEEE International Conference on Robotics and Automation , 2002, pp. 1398–1403. DOI: 10.1109/ROBOT.2002.1014739

    arXiv 2002

  7. [7]

    Probabilistic movement primitives,

    A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann, “Probabilistic movement primitives,” Advances in Neu- ral Information Processing Systems , vol. 26, 2013

    work page 2013

  8. [8]

    Luo et al

    J. Luo et al. , SERL: A software suite for sample- efficient robotic reinforcement learning , 2024. arXiv: 2401.16013

    arXiv 2024

Show all 53 references
  1. [10]

    Scalable global optimization via local Bayesian optimization,

    D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek, “Scalable global optimization via local Bayesian optimization,” in Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  2. [11]

    Local policy search with Bayesian optimization,

    S. M ¨uller, A. von Rohr, and S. Trimpe, “Local policy search with Bayesian optimization,” in Advances in Neural Information Processing Systems , vol. 34, 2021, pp. 20 708–20 720

    2021

  3. [12]

    Cautious Bayesian optimization for efficient and scal- able policy search,

    L. P. Fr ¨ohlich, M. N. Zeilinger, and E. D. Klenske, “Cautious Bayesian optimization for efficient and scal- able policy search,” in Learning for Dynamics and Control, 2021, pp. 227–240

    2021

  4. [13]

    High-dimensional Bayesian optimization with sparse axis-aligned sub- spaces,

    D. Eriksson and M. Jankowiak, “High-dimensional Bayesian optimization with sparse axis-aligned sub- spaces,” in Uncertainty in Artificial Intelligence , 2021, pp. 493–503

    2021

  5. [14]

    Lo- cal Bayesian optimization via maximizing probability of 15 descent,

    Q. Nguyen, K. Wu, J. R. Gardner, and R. Garnett, “Lo- cal Bayesian optimization via maximizing probability of 15 descent,” in Advances in Neural Information Processing Systems, 2022

    2022

  6. [15]

    Are random decom- positions all we need in high dimensional Bayesian optimisation?

    J. K. Ziomek and H. B. Ammar, “Are random decom- positions all we need in high dimensional Bayesian optimisation?” In International Conference on Machine Learning, 2023, pp. 43 347–43 368

    2023

  7. [16]

    Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task,

    S. James, A. J. Davison, and E. Johns, “Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task,” in Conference on Robot Learning, 2017, pp. 334–343

    2017

  8. [17]

    Sim-to-real transfer of robotic control with dynamics randomization,

    X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in IEEE International Con- ference on Robotics and Automation , 2018, pp. 3803–

    2018

  9. [18]

    Learning dexterous in-hand manipulation,

    M. Andrychowicz et al. , “Learning dexterous in-hand manipulation,” The International Journal of Robotics Research, vol. 39, no. 1, pp. 3–20, 2020. DOI: 10.1177/ 0278364919887447

    2020

  10. [19]

    Learning invariant feature spaces to transfer skills with reinforcement learning,

    A. Gupta, C. Devin, Y . Liu, P. Abbeel, and S. Levine, “Learning invariant feature spaces to transfer skills with reinforcement learning,” in International Conference on Learning Representations, 2017

    2017

  11. [20]

    Sim-to-real robot learning from pixels with progressive nets,

    A. A. Rusu, M. Ve ˇcer´ık, T. Roth ¨orl, N. Heess, R. Pascanu, and R. Hadsell, “Sim-to-real robot learning from pixels with progressive nets,” in Conference on Robot Learning, 2017, pp. 262–270

    2017

  12. [21]

    DIRL: Domain-invariant representation learning for sim-to-real transfer,

    A. Tanwani, “DIRL: Domain-invariant representation learning for sim-to-real transfer,” in Conference on Robot Learning, 2021, pp. 1558–1571

    2021

  13. [22]

    Gaussian process optimization in the bandit setting: No regret and experimental design,

    N. Srinivas, A. Krause, S. Kakade, and M. Seeger, “Gaussian process optimization in the bandit setting: No regret and experimental design,” in International Con- ference on Machine Learning , 2010, pp. 1015–1022

    2010

  14. [24]

    Automatic LQR tuning based on Gaussian process global optimization,

    A. Marco, P. Hennig, J. Bohg, S. Schaal, and S. Trimpe, “Automatic LQR tuning based on Gaussian process global optimization,” in IEEE International Conference on Robotics and Automation , 2016, pp. 270–277. DOI: 10.1109/ICRA.2016.7487144

    2016

  15. [25]

    Automatic tuning for data-driven model predictive control,

    W. Edwards, G. Tang, G. Mamakoukas, T. Murphey, and K. Hauser, “Automatic tuning for data-driven model predictive control,” in IEEE International Conference on Robotics and Automation , 2021, pp. 7379–7385. DOI: 10.1109/ICRA48506.2021.9562025

    2021

  16. [26]

    Bayesian optimisation for robust model predictive control un- der model parameter uncertainty,

    R. Guzman, R. Oliveira, and F. Ramos, “Bayesian optimisation for robust model predictive control un- der model parameter uncertainty,” in IEEE Interna- tional Conference on Robotics and Automation , 2022, pp. 5539–5545. DOI: 10 . 1109 / ICRA46639 . 2022 . 9812406

    2022

  17. [27]

    Computing the racing line using Bayesian optimization,

    A. Jain and M. Morari, “Computing the racing line using Bayesian optimization,” in IEEE Conference on Decision and Control , 2020, pp. 6192–6197. DOI: 10. 1109/CDC42340.2020.9304147

    2020

  18. [28]

    Learning by demonstration and robust control of dexterous in-hand robotic manipu- lation skills,

    G. Solak and L. Jamone, “Learning by demonstration and robust control of dexterous in-hand robotic manipu- lation skills,” in IEEE/RSJ International Conference on Intelligent Robots and Systems , 2019, pp. 8246–8251. DOI: 10.1109/IROS40897.2019.8967567

    2019

  19. [29]

    Deep black-box reinforcement learning with movement primitives,

    F. Otto, O. Celik, H. Zhou, H. Ziesche, V . A. Ngo, and G. Neumann, “Deep black-box reinforcement learning with movement primitives,” in Conference on Robot Learning, 2023, pp. 1244–1265

    2023

  20. [30]

    PILCO: A model- based and data-efficient approach to policy search,

    M. Deisenroth and C. E. Rasmussen, “PILCO: A model- based and data-efficient approach to policy search,” in International Conference on Machine Learning , 2011, pp. 465–472

    2011

  21. [31]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models,

    K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” in Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  22. [32]

    Sample-efficient reinforcement learning with stochastic ensemble value expansion,

    J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee, “Sample-efficient reinforcement learning with stochastic ensemble value expansion,” in Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  23. [33]

    When to trust your model: Model-based policy optimization,

    M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” in Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  24. [34]

    Data efficient reinforcement learning for legged robots,

    Y . Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V . Sindhwani, “Data efficient reinforcement learning for legged robots,” in Conference on Robot Learning, 2020, pp. 1–10

    2020

  25. [35]

    Learning modular robot control policies,

    J. Whitman, M. Travers, and H. Choset, “Learning modular robot control policies,” IEEE Transactions on Robotics, vol. 39, no. 5, pp. 4095–4113, 2023. DOI: 10.1109/TRO.2023.3284362

    2023

  26. [36]

    Garnett, Bayesian Optimization

    R. Garnett, Bayesian Optimization. Cambridge Univer- sity Press, 2023

    2023

  27. [37]

    Automatic gait optimization with Gaussian process regression,

    D. J. Lizotte, T. Wang, M. H. Bowling, and D. Schu- urmans, “Automatic gait optimization with Gaussian process regression,” in International Joint Conferences on Artificial Intelligence , vol. 7, 2007, pp. 944–949

    2007

  28. [39]

    Virtual vs. real: Trading off sim- ulations and physical experiments in reinforcement learning with Bayesian optimization,

    A. Marco et al. , “Virtual vs. real: Trading off sim- ulations and physical experiments in reinforcement learning with Bayesian optimization,” in IEEE Interna- tional Conference on Robotics and Automation , 2017, pp. 1557–1563. DOI: 10.1109/ICRA.2017.7989186

    2017

  29. [40]

    Local Bayesian optimization of motor skills,

    R. Akrour, D. Sorokin, J. Peters, and G. Neumann, “Local Bayesian optimization of motor skills,” in In- ternational Conference on Machine Learning , vol. 70, 2017, pp. 41–50

    2017

  30. [41]

    Domain randomization for transferring deep neural networks from simulation to the real world,

    J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” 16 in IEEE/RSJ International Conference on Intelligent Robots and Systems , 2017, pp. 23–30. DOI: 10.1109/ IROS.2...

    2017

  31. [42]

    Robot learning from randomized simulations: A review,

    F. Muratore, F. Ramos, G. Turk, W. Yu, M. Gienger, and J. Peters, “Robot learning from randomized simulations: A review,” Frontiers in Robotics and AI , vol. 9, Apr

  32. [43]

    Rasmussen and C

    C. Rasmussen and C. Williams, Gaussian Processes for Machine Learning . MIT Press, 2006. DOI: mitpress / 3206.001.0001

    2006

  33. [44]

    B. T. Polyak, Introduction to Optimization . Optimiza- tion Software, Inc., 1987

    1987

  34. [45]

    J. Watt, R. Borhani, and A. K. Katsaggelos, Machine learning refined: Foundations, algorithms, and applica- tions. Cambridge University Press, 2020

    2020

  35. [46]

    Multi- information source optimization,

    M. Poloczek, J. Wang, and P. Frazier, “Multi- information source optimization,” in Advances in Neu- ral Information Processing Systems , vol. 30, 2017

    2017

  36. [47]

    Efficient global optimization of expensive black-box functions,

    D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient global optimization of expensive black-box functions,” Journal of Global Optimization, vol. 13, no. 4, pp. 455– 492, 1998

    1998

  37. [48]

    Simple random search of static linear policies is competitive for rein- forcement learning,

    H. Mania, A. Guy, and B. Recht, “Simple random search of static linear policies is competitive for rein- forcement learning,” in Advances in Neural Information Processing Systems, vol. 31, 2018

    2018

  38. [49]

    Entropy search for information-efficient global optimization,

    P. Hennig and C. J. Schuler, “Entropy search for information-efficient global optimization,” Journal of Machine Learning Research , vol. 13, no. 6, 2012

    2012

  39. [50]

    F. Otto, O. Celik, D. Roth, and H. Zhou, Fancy gym, Au- tonomous Learning Robots Lab (ALR) at KIT. [Online]. Available: https://github.com/ALRhub/fancy gym

  40. [51]

    Deep neural networks for improved, impromptu trajectory tracking of quadrotors,

    Q. Li, J. Qian, Z. Zhu, X. Bao, M. K. Helwa, and A. P. Schoellig, “Deep neural networks for improved, impromptu trajectory tracking of quadrotors,” in IEEE International Conference on Robotics and Automation , 2017, pp. 5183–5189. DOI: 10 . 1109 / ICRA . 2017 . 7989607

    2017

  41. [52]

    Learning from demonstration,

    S. Schaal, “Learning from demonstration,” in Advances in Neural Information Processing Systems, vol. 9, 1996

    1996

  42. [53]

    Dynamical movement primitives: Learning attractor models for motor behaviors,

    A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical movement primitives: Learning attractor models for motor behaviors,” Neural Compu- tation, vol. 25, no. 2, pp. 328–373, 2013. DOI: 10.1162/ NECO a 00393

    2013

  43. [54]

    The be- havior and convergence of local Bayesian optimization,

    K. Wu, K. Kim, R. Garnett, and J. R. Gardner, “The be- havior and convergence of local Bayesian optimization,” in Advances in Neural Information Processing Systems , vol. 36, 2023

    2023

  44. [2022]

    DOI: 10.3389/frobt.2022.799893

    2022

  45. [3810]

    DOI: 10.1109/ICRA.2018.8460528

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.