{"id":"59a59606-7169-488c-8548-3cbbf0d56af2","arxiv_id":"2411.14246","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An adaptive Bayesian policy-search method (HCI-GIBO/S-HCI-GIBO) uses simulator data to reduce real-robot queries while aiming for high-confidence policy improvements.","lead":"This paper introduces a Bayesian optimization method that fine-tunes robot control policies using both real-robot experiments and a simulator, deciding when simulator data is sufficient. It reports faster learning with fewer robot interactions, shown on synthetic benchmarks and a physical Franka arm.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The improvement guarantee in Theorem 1 depends on the Lipschitz constant L, but the experiments never specify or estimate L; without it the COMMITMENT condition cannot be evaluated, so the reported results do not instantiate the central guarantee.","rationale":"The paper's central claim is a per-update improvement guarantee with high probability. The theoretical result (Lemma 1 and Theorem 1) is logically sound: if the gradient is Lipschitz with constant L and the posterior condition (19) holds, improvement follows with probability at least α. However, the algorithm cannot be instantiated without a numerical value for L, and the experimental sections provide none. This is not merely a presentational issue: the COMMITMENT function is the mechanism that replaces the fixed query budget M in GIBO, so the claimed data-efficiency and guarantee both hinge on this probability being computable. The omission is visible in the robot experiment (Section VIII-B-b) and in the synthetic benchmarks (Section VII-A), where objectives are sampled from a GP and L varies per sample. The paper's Limitations paragraph acknowledges that performance depends on informed priors, but it does not address the operational question of how L is set. The reader's weakest_assumption identifies the same issue, and the proposed test—checking the code and re-running with scaled L—would settle whether the reported results actually validate the guarantee or merely reflect an implicit choice of L. Thus the conditional verdict is appropriate; no change is needed.","tokens_in":21110,"tokens_out":11413,"duration_ms":108750,"concrete_test":"Inspect the released implementation (github.com/Data-Science-in-Mechanical-Engineering/hci-gibo) to find how COMMITMENT is computed and what value or estimator of L is used. Then re-run the real-robot experiment with L multiplied by 2 and by 0.5, keeping α=0.95 and η=0.2, and measure the number of real-robot queries to reach 80% of the maximal reward and the empirical frequency of per-update improvement. If the query count changes materially or the improvement frequency drops below α, the central guarantee is sensitive to an unspecified hyperparameter and the paper must state how L is set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 1 assumes the true gradient is Lipschitz with known constant L, and Theorem 1's improvement probability (Eq. 19) is computed with L in the threshold L η ||μ||/2. To run HCI-GIBO, the algorithm must set L. Section VIII-B-b lists the hyperparameters α=0.95 and η=0.2 for the robot experiment but never mentions L, nor does any other experimental section. The same gap appears in the synthetic within-model benchmarks: objectives are sampled from a GP, so L is sample-dependent, yet no L value or estimation procedure is reported. Consequently, the COMMITMENT function (19) is not well-defined in the implementation, and the claim that the algorithm 'queries until it can guarantee improvement with high probability' is not verified by the experiments. If L was set implicitly (e.g., to 1) or estimated ad hoc, the reported data-efficiency numbers are conditional on that choice; a mismatch between the chosen L and the true Lipschitz constant breaks the high-probability improvement guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes HCI-GIBO, a local black-box policy-search algorithm based on Gradient Information with Bayesian Optimization (GIBO), and S-HCI-GIBO, a multi-fidelity extension that uses a simulator as an additional information source. The key idea is to replace GIBO's fixed number of inner-loop queries with a commitment rule derived from a Gaussian-process posterior over the objective's gradient: the algorithm queries until the probability that a gradient step improves the objective exceeds a threshold alpha (Theorem 1, Eq. (19)). S-HCI-GIBO models robot and simulator returns with a dual-source GP kernel and switches from simulator to real-robot queries when the marginal gradient information of a simulator query falls below beta. The paper reports synthetic within-model benchmarks up to 52 dimensions, fine-tuning experiments on three 'fancy gym' environments, and a real Franka Emika robot experiment for trajectory tracking of a pendulum, where S-HCI-GIBO is claimed to reach 80% of maximal reward with fewer than half the robot queries of HCI-GIBO.","tokens_in":21331,"tokens_out":10565,"duration_ms":97853,"significance":"The proposed commitment rule is a principled way to adapt the number of queries per gradient step and could improve data efficiency in black-box policy search; the multi-fidelity formulation with a binary information-source indicator is a natural extension of GIBO. The paper is strengthened by open-source code, a released ROS package, and a real-hardware evaluation with an imperfect simulator. However, the theoretical guarantee is conditional on the Lipschitz constant L and on the GP model being well specified, and the experimental sections do not instantiate L; there is also an inconsistency in the definition of the reality gap and the resulting kernel. These issues do not invalidate the algorithmic idea but must be fixed before the central claims can be accepted as stated.","major_comments":[{"comment":"The COMMITMENT criterion depends on the Lipschitz constant L, but L is never specified or estimated in any experiment. Theorem 1's guarantee therefore is not instantiated by the reported results; the claim that the algorithm 'queries until it can guarantee that the update step will improve the function with high probability' (Section V-B) is not verified. The authors should either fix a conservative L with justification, estimate L from the GP posterior or sample-path bounds, or report a sensitivity analysis over L; without one of these, Eq. (19) is not evaluable from the paper.","section":"Section V-B, Eq. (19); Sections VII-A and VIII-B-b"},{"comment":"The model definition is inconsistent. Eq. (3) sets f_sim = f + f_gap, but the text immediately after defines f_gap = f - f_sim; these differ by a sign. Moreover, kernel (21) with k_delta(IS,IS*) = IS*IS* gives Cov(f_sim(theta), f_sim(theta')) = Kf and Cov(f(theta), f(theta')) = Kf + Km, which corresponds to f = f_sim + f_gap, not f_sim = f + f_gap. Please correct the definitions and state precisely which function is the 'common' GP and which is the source-specific discrepancy; this is essential for reproducing the dual-information-source model.","section":"Section II, Eq. (3) and Assumption 1; Section V-C-2, Eq. (21)"},{"comment":"The treatment of normalized gradient steps is ambiguous. Eq. (19) is derived for an update theta - eta*mu, with threshold (L/2)*eta*||mu||; if the descent direction is normalized to mu/||mu||, the threshold in (19) becomes (L/2)*eta, not (L/2)*eta*||mu||. The distribution in (20) is the correct one for a normalized direction, but the text does not state which form of (19) is used in the experiments. Please specify the exact update rule and corresponding threshold used in each experimental section.","section":"Section V-B, Remark 1 and Eqs. (19)-(20)"}],"minor_comments":[{"comment":"The vector nu is called a 'descent direction', but condition (10) is a positive inner product, which corresponds to an ascent direction if the update is theta - eta*nu; please use 'update direction' or clarify the sign convention.","section":"Section V-B, Lemma 1"},{"comment":"The 'less than half' claim is based on five trials; please report per-trial query counts or confidence intervals for the reward levels so the reader can assess the variability of this headline number.","section":"Section VIII-B-c and Table I"},{"comment":"The caption states that shaded regions on the x-axis show standard deviation, but the plot appears to show shaded regions around the curves; the caption and the plot should be aligned.","section":"Fig. 12"},{"comment":"The budget statement for S-HCI-GIBO is ambiguous because simulator evaluations are excluded from the x-axis of Fig. 6; please state the total simulator budget explicitly.","section":"Section VII-A"}],"recommendation":"major_revision","confidential_remarks":"The paper's algorithmic contribution is promising and the empirical direction is relevant for the journal, but the missing specification of L and the inconsistency in the multi-fidelity model definition are load-bearing for the main claims. Both are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper does something useful, but the headline guarantee is under-specified in the experiments. It replaces GIBO's fixed M queries with an adaptive stop rule based on posterior probability of improvement, and adds a second information source (simulator) through the Marco et al. kernel. That combination is new, modest but real, and the empirical story is consistent: HCI-GIBO beats GIBO on synthetic benchmarks, and the real-robot pendulum experiment shows S-HCI-GIBO needing less than half the robot evaluations to reach 80% of max reward.\n\nWhat's done well: Theorem 1 is a correct sufficient condition, the derivation is clean, and the code is posted. The synthetic ablations isolate the effect of the commitment rule and the simulator. The real hardware experiment is a genuine deployment with an imperfect simulator, and the sim-only control shows why the real robot still matters.\n\nSoft spots, in proportion: The biggest issue, which the stress-test note nails, is that L appears in the commitment condition (19) and Theorem 1, but no experimental section specifies L or gives an estimation procedure. So the 'guarantee' is not instantiated. The algorithm could be run with L set implicitly, but then the probability statement is conditional on an arbitrary L. This is not fatal to the empirical method, but it is a real gap between theory and practice. The authors should either estimate L from the GP or treat it as a hyperparameter and report it.\n\nAlso, the synthetic benchmarks are within-model: the objectives are drawn from the same GP prior the algorithm assumes. That is standard practice in BO theory papers, but it means the synthetic numbers are best-case. The real-robot result is five trials on one task, so treat the 'half the queries' number as preliminary. The fancy-gym experiments are on simulated tasks, not hardware.\n\nThe citation pattern looks fine; the self-citation to GIBO is appropriate since that's their own baseline and the adaptive rule is an independent derivation.\n\nBottom line: worth refereeing. The method is simple, reproducible, and likely useful for fine-tuning movement primitives. A serious referee should press on L and ask for more hardware data, but the paper is not fatally flawed.","headline":"Useful extension of GIBO with an adaptive stop rule and a simulator channel, but the central improvement guarantee is not instantiated in experiments because the Lipschitz constant L is never set or estimated.","tokens_in":21873,"tokens_out":2372,"would_cite":true,"duration_ms":21757,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By keeping a posterior over the objective's gradient and querying until a policy update is statistically guaranteed to improve, this paper makes simulator-assisted black-box policy search work on a real robot with fewer than half the…","keywords":["Bayesian optimization","black-box policy search","sim-to-real transfer","multi-fidelity optimization","Gaussian processes","movement primitives","robot learning","derivative-free optimization"],"falsifier":"Run the pendulum tracking task of Section VIII-B with $\\alpha=0.95$ and $\\eta=0.2$ while setting $L$ too small (e.g., by a factor of ten); if the measured frequency of reward-decreasing updates exceeds $1-\\alpha=0.05$, the guarantee fails in the regime the paper leaves unspecified. Even without a new run, counting per-update successes in the reported data would settle it: the paper plots average reward but does not report the fraction of updates that actually decreased reward.","tokens_in":20902,"feed_emoji":"🤖","tokens_out":10299,"duration_ms":85771,"temperature":0.7,"pith_summary":"The paper tries to make black-box policy search—tuning a robot controller by observing only the reward of trial runs—practical for real hardware, where every evaluation is expensive. Its proposal, HCI-GIBO, keeps collecting data until the posterior belief over the objective's gradient makes it highly probable that the next policy update will improve performance, instead of fixing the number of evaluations per update in advance. S-HCI-GIBO adds a second, cheaper information source: an imperfect simulator that is queried first and only abandoned when its data stops shrinking gradient uncertainty. The authors demonstrate on a real robot manipulator balancing a pendulum that the simulator-aided version reaches 80 percent of the maximal reward with less than half the real-robot queries of the simulator-free version. The significance would be that fast adaptation on physical robots, previously limited to low-dimensional or heavily engineered settings, becomes feasible for higher-dimensional movement-primitive policies.","feed_headline":"Robot learning uses half the real-world trials with simulator help","feed_subtitle":"A high-confidence Bayesian optimizer decides when simulator data is enough, cutting real-robot queries by over half.","key_machinery":"The carrying object is the derivative Gaussian process, a joint GP over zeroth-order observations and the gradient at the current best policy $\\hat\\theta^*$; conditioning it on the data gives a closed-form Gaussian posterior on the gradient, so the gradient-information acquisition function can score candidate query points by how much they shrink the trace of the gradient covariance without knowing the outcome. The descent rule uses the posterior mean as the step direction, and the stopping rule uses Lemma 1: a step of size $\\eta$ in the normalized direction $\\mu/\\|\\mu\\|$ improves the cost whenever the true gradient's projection onto that direction exceeds $\\tfrac{L}{2}\\eta\\|\\mu\\|$, where $L$ bounds the gradient's Lipschitz constant. Theorem 1 replaces the true gradient by the Gaussian belief $\\nabla F(\\hat\\theta^*) \\sim \\mathcal{N}(\\mu,\\Sigma)$, so the probability in condition (19) is a tail probability of a univariate normal, making the commitment test computable. The simulator extension is carried by a dual-source kernel $K(\\bar X, x^*) = K_f(\\theta,\\hat\\theta^*) + k_\\delta(\\mathrm{IS}, \\mathrm{IS}^*)K_m(\\theta,\\hat\\theta^*)$, which couples robot and simulator observations, plus a switching rule that moves from simulator to robot when consecutive simulator queries reduce the robot-gradient uncertainty by less than the threshold $\\beta$.","core_discovery":"On its own terms, the paper's claim is that a policy-update rule can be made both data-efficient and trustworthy: query the objective only as long as needed for the probability that the gradient step improves the function to reach a user-set confidence alpha, and then step. Theorem 1 turns Lemma 1's deterministic descent condition into a probabilistic one by replacing the true gradient with the Gaussian posterior over the gradient, so the commitment condition is computable in closed form. The S-HCI-GIBO extension says that before spending real-robot queries, the optimizer should exhaust the gradient information available from a biased simulator, using a dual-source derivative GP whose kernel couples the two sources, and switch to the robot only when another simulator query would reduce the robot-gradient uncertainty by less than a threshold beta. The real-robot pendulum experiment is then presented as evidence that this scheme learns an impromptu Lissajous tracking task from a zero-initialized dynamic movement primitive, and that the simulator-aided run reaches 80 percent of the maximum reward with less than half the real-robot evaluations of HCI-GIBO alone (58 vs 132 evaluations in Table I).","pith_inferences":["The same query-until-confident gradient rule could be applied to other expensive feedback channels, such as human preference ratings, where each query is a person's judgment; the only ingredient needed is a GP prior on the objective and its gradient.","A testable extension would be to estimate $L$ online from the GP's posterior and shrink the step size $\\eta$ when the local surface is flat; the paper fixes $\\eta$ and leaves $L$ unset, so confidence $\\alpha$ and actual improvement frequency can drift apart.","The dual-source kernel's switching rule assumes one static simulator; extending the same formula to multiple simulators of different fidelity, or to a simulator whose bias drifts over time, would follow naturally but is not analyzed here."],"forward_implications":["With the reported confidence level $\\alpha=0.95$ on the pendulum hardware, each policy update should decrease reward with probability at most 0.05, so the per-update failure rate is a checkable prediction of the method.","Simulator data is not just a warm start: on the 24-dimensional pendulum task, reaching 80 percent of the maximal reward takes 58 real-robot evaluations with S-HCI-GIBO versus 132 without, a reduction by more than half.","The commitment rule replaces the fixed per-update query count of GIBO, so the number of evaluations per update adapts to the local landscape; the synthetic benchmarks show the advantage growing in 20- to 52-dimensional domains.","The method is a local fine-tuner, not a global planner: it assumes a reasonable initial policy (from a movement primitive or pre-trained RL) and improves it, so its success on hardware does not transfer to cold-start global discovery."],"supporting_citations":[{"why":"Supplies the gradient-information acquisition function and derivative GP posterior that HCI-GIBO builds on.","marker":"[11]"},{"why":"Supplies the dual-information-source GP coupling that S-HCI-GIBO adapts from global to local search.","marker":"[39]"},{"why":"Motivates the multi-fidelity kernel that lets robot and simulator observations be fused in one model.","marker":"[46]"},{"why":"A local BO method using probability of descent; the paper's commitment rule is positioned against its fixed-query approach.","marker":"[14]"},{"why":"The confidence-region baseline CRBO and the motivation for local over global Bayesian optimization.","marker":"[12]"},{"why":"Provides the pre-trained movement-primitive policies and the black-box RL baseline used in the fine-tuning experiments.","marker":"[29]"},{"why":"The probabilistic movement primitive policy representation used in the simulated fine-tuning tasks.","marker":"[7]"},{"why":"The dynamical movement primitive representation used in the hardware pendulum experiment.","marker":"[53]"}],"fun_headline_variants":["Simulator-aided policy search halves real-robot queries","Robot learning with imperfect sim uses half the real trials","Dual-source optimizer reduces real-robot evaluations by half","Confidence-guaranteed robot policy updates use fewer real trials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee that each update improves with probability at least $\\alpha$ requires the gradient Lipschitz constant $L$ from Assumption 1 to be known and set correctly, yet the experimental sections report $\\alpha$ and $\\eta$ but never $L$, so the stated confidence is not instantiated in the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Simulator-aided policy search halves real-robot queries","Robot learning with imperfect sim uses half the real trials","Dual-source optimizer reduces real-robot evaluations by half","Confidence-guaranteed robot policy updates use fewer real trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2902,"prompt_tokens":935,"completion_tokens":1967,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1900}},"tokens_in":551,"tokens_out":1967,"duration_ms":14821,"temperature":1.0,"reasoning_tokens":1900,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:23:03.687008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pendulum tracking task of Section VIII-B with $\\alpha=0.95$ and $\\eta=0.2$ while setting $L$ too small (e.g., by a factor of ten); if the measured frequency of reward-decreasing updates exceeds $1-\\alpha=0.05$, the guarantee fails in the regime the paper leaves unspecified. Even without a new run, counting per-update successes in the reported data would settle it: the paper plots average reward but does not report the fraction of updates that actually decreased reward.","supporting_citations":[{"cited_title":"Local policy search with Bayesian optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-information acquisition function and derivative GP posterior that HCI-GIBO builds on."},{"cited_title":"Multi- information source optimization,","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-fidelity kernel that lets robot and simulator observations be fused in one model."},{"cited_title":"Lo- cal Bayesian optimization via maximizing probability of 15 descent,","cited_arxiv_id":null,"evidence_quote":"A local BO method using probability of descent; the paper's commitment rule is positioned against its fixed-query approach."},{"cited_title":"Cautious Bayesian optimization for efficient and scal- able policy search,","cited_arxiv_id":null,"evidence_quote":"The confidence-region baseline CRBO and the motivation for local over global Bayesian optimization."},{"cited_title":"Deep black-box reinforcement learning with movement primitives,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained movement-primitive policies and the black-box RL baseline used in the fine-tuning experiments."},{"cited_title":"Probabilistic movement primitives,","cited_arxiv_id":null,"evidence_quote":"The probabilistic movement primitive policy representation used in the simulated fine-tuning tasks."},{"cited_title":"Dynamical movement primitives: Learning attractor models for motor behaviors,","cited_arxiv_id":null,"evidence_quote":"The dynamical movement primitive representation used in the hardware pendulum experiment."}],"review_version":1}