{"id":"48e3c4da-cdda-425a-9a72-aaad387fca63","arxiv_id":"2507.02256","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"URDP couples LLM-based reward component design with uncertainty-weighted Bayesian optimization, reporting better reward quality and efficiency than Eureka and Text2Reward on three benchmarks.","lead":"This paper introduces URDP, a framework that combines large language models with Bayesian optimization to design reward functions for reinforcement learning. It reports higher reward quality and lower simulation costs than prior methods across 35 robotic tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Uncertainty score U(r) is the load-bearing input to both filtering and UABO, yet no evidence links it to reward quality; propose a positive-control comparison.","rationale":"The paper proposes a decoupled reward design process whose headline gains (Table 1 and Table 2) depend on two components: uncertainty screening that avoids simulations and uncertainty-weighted BO that accelerates search. Both components consume the same scalar U(r), so the assumption that U(r) is a reliable quality signal is uniquely load-bearing. The reader identified this same assumption as the weakest assumption; I agree. I examined the appendices and experiments for independent support: no oracle, random-baseline, or correlation analysis of U(r) exists; Abl-1 only compares URDP with and without uncertainty on aggregate NOE; Disc-2 reports high-uncertainty components are often useful, which does not refute the screening rationale but highlights the need for direct validation. The proof in App. F.2 is conditional on U(r) identifying irrelevant dimensions, which is exactly the unverified assumption. The missing seeds/error bars amplify the risk: Table 6 shows per-task NOE/NLC where URDP sometimes uses fewer evaluations than Eureka while achieving a higher score, so the 'identical simulation budget' comparison in Section 5.3 needs precise definition. I also checked the reported 52.4%/46.6% figures against Table 6 aggregated values; the claim appears to be an aggregation over tasks, but the appendix does not give the exact formula, so it is worth a quick audit, though not the primary concern. None of this makes the paper unsound; the empirical results are internally consistent modulo missing error bars, and the decoupled architecture is a reasonable contribution even if uncertainty is not validated. That said, the paper explicitly frames uncertainty as the mechanism for both efficiency and performance (abstract and contributions), so a conditional acceptance with a request for a direct validation study is appropriate.","tokens_in":27097,"tokens_out":3163,"duration_ms":30225,"concrete_test":"Run a controlled comparison on the Isaac/Dexterity/ManiSkill2 subsets where URDP's uncertainty screening is replaced by (a) random screening of the same number of components and (b) oracle screening based on final simulation performance. If URDP's uncertainty selection does not beat random selection on a paired-seed comparison, the claim that similarity-based uncertainty identifies ineffective components is unsupported. Additionally, plot correlation between U(r) and the marginal effect on final policy performance across all 35 tasks; if the correlation is nonpositive, then U(r) should not weight the kernel or acquisition function.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 6: URDP outperforms existing methods in reward quality and design efficiency; Section 5.3: 3.424 vs 1.607 HNS on Isaac, 0.675/0.792 vs 0.466/0.449 SR) depends on two mechanisms: (i) uncertainty screening pre-simulation to discard redundant reward components, and (ii) uncertainty-weighted kernel and uEI acquisition in Section 4.3. Both mechanisms rest on the unvalidated assumption that the similarity-based uncertainty score U(r) in Eq. 2 correlates with reward quality or usefulness. Section 4.2 asserts that high-consistency outputs are 'well-internalized, task-specific knowledge' and that divergent outputs 'demand more refinement,' but no experiment compares URDP's uncertainty ranking against an oracle or random selection, or against simulation-based ground truth. If high-uncertainty components are merely lexically/semantically novel, then (a) filtering may discard useful components (Section 5.4 Abl-1 shows only aggregate NOE savings vs. URDP w.o. Uncertainty, not whether filtered components were actually ineffective), and (b) the anisotropic kernel and the uEI penalty w(theta) in Eq. 8 may direct search away from high-uncertainty dimensions that are actually important, causing the reported efficiency and performance gains to come from the decoupled BO alone, not from uncertainty information. Disc-2 (Section 5.5) even shows that high-uncertainty components often contribute positively, which directly tensions with the claim that divergent outputs 'demand more refinement.' The proof in App. F.2 relies on the limiting assumption that f is constant along d-m directions and that U(r) correctly identifies those directions; without empirical support that U(r) tracks actual dimension importance, the theorem does not connect to the algorithm's success.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes URDP, an LLM-based automated reward design framework that decouples reward component generation from reward-intensity hyperparameter optimization. In the outer loop, LLM-generated reward component samples are filtered using a self-consistency-derived uncertainty score U(r), and in the inner loop an uncertainty-aware Bayesian optimization (UABO) tunes reward intensities. The authors evaluate URDP on 35 tasks across IsaacGym, Dexterity, and ManiSkill2, reporting improved success rates and human-normalized scores relative to Eureka, Text2Reward, human-designed, and sparse rewards, with reduced numbers of simulation episodes and LLM calls. The main claimed contributions are simulation-free filtering of ineffective reward components, discovery of novel reward components via uncertainty, and an uncertainty-weighted kernel and acquisition function that accelerate BO convergence.","tokens_in":27379,"tokens_out":5861,"duration_ms":66898,"significance":"If the central claims held, the paper would make a useful contribution to automated reward design: the bi-level architecture is a sensible response to known limitations of pure LLM evolutionary search, and the broad 35-task evaluation is more extensive than in many prior works in this area. The paper also includes useful ablations (uncertainty filtering, decoupled optimization, UABO vs. LLM optimizers) and a good-faith discussion of limitations and LLM-agnostic behavior, which are strengths. However, the central empirical claims are currently under-supported: the uncertainty score is not validated as a predictor of reward-component quality, the headline performance gains are not reproduced by the reported tables, and several mechanisms appear internally inconsistent. The framework is plausible, but the evidence as presented is not yet sufficient to establish the claimed advances.","major_comments":[{"comment":"The uncertainty score U(r_i) is the load-bearing input to both the filtering step and UABO, but no experiment validates the assumption that similarity-based consistency correlates with reward-component quality. Section 4.2 asserts that high-consistency outputs are 'well-internalized, task-specific knowledge' and that divergent outputs 'demand more refinement,' yet no comparison is made against random filtering, against keeping all components, or against an oracle selection based on simulation. Moreover, Disc-2 (Section 5.5) shows that high-uncertainty components (U > 0.9) are often novel and contribute positively to reward shaping, which creates a direct tension with the use of high uncertainty to penalize exploration in Eq. (8). Please add a positive-control experiment: for a subset of tasks, compare URDP with (a) random component filtering at the same removal rate, (b) no filtering, and (c) an oracle that keeps components whose leave-one-out simulation impact is positive, and report the correlation between U(r) and per-component impact.","section":"Section 4.2, Eq. (2); Section 5.5 Disc-2"},{"comment":"The headline performance claims are not supported by the reported numbers. The text states a 132% gain over Eureka on Isaac, but Table 2 gives (3.424 - 1.607)/1.607 ≈ 113%, not 132%. More seriously, Tables 6 and 7 report inconsistent values for the same tasks: for Ant, Eureka has HNS 1.527 with NOE 112 in Table 6, but HNS 0.828 with NOE 48 in Table 7, while URDP has HNS 1.556 in both tables; this suggests the comparison is being made at different simulation budgets without explanation. All reported results appear to be single runs, with no seed information, standard deviations, or significance tests. Because the central claim is that URDP improves both quality and efficiency, multi-seed runs with error bars and a clear statement of the evaluation budget for each method are required.","section":"Section 5.3, Tables 2, 6, and 7"},{"comment":"The ablation shows that removing uncertainty screening increases NOE while achieving a comparable success rate, but it does not establish that the filtered components were ineffective. The observed savings could come simply from deduplicating lexically redundant samples, or from discarding useful components whose absence is compensated by other parts of the framework. To support the claim that the uncertainty signal identifies ineffective components, the authors should report, for at least a subset of tasks, the fraction of filtered components that would have made a positive contribution if kept (measured by simulation), and compare against random filtering at the same removal rate.","section":"Section 5.4 Abl-1"},{"comment":"The uEI acquisition function penalizes movement away from the incumbent more strongly along dimensions with high U(r_j), on the rationale that high-uncertainty directions should be exploited rather than explored. However, Section 5.5 Disc-2 reports that high-uncertainty reward components are frequently novel and that removing them degrades performance, implying that these directions are important. This is an internal inconsistency: the same signal is used both to allocate additional inner-loop iterations to high-uncertainty samples (Section 4.2) and to restrict hyperparameter exploration along those dimensions. The paper does not explain this conflict, and the reported efficiency gains may therefore be attributable to the decoupled BO alone rather than to the uncertainty weighting. Please run an ablation in which the weights in Eq. (8) are replaced by random or constant values while all other components of UABO are unchanged.","section":"Section 4.3, Eq. (8); Section 5.5 Disc-2"},{"comment":"The convergence proof for uEI assumes that the function f depends on only m variables and sets the weights λ_j to 0 for the relevant dimensions and ∞ for the irrelevant ones. The implemented algorithm, however, uses data-dependent weights U(r_j) that are not 0/∞ weights and for which no effective-dimension assumption is established. The stated rate of n^{-(ν∧1)/m} therefore does not follow for the actual uEI strategy. Please either provide a convergence result for the actual weighting scheme under explicit regularity conditions, or clearly present the existing proof as a heuristic motivation rather than a theorem about the implemented algorithm.","section":"Appendix F, Theorem 2"}],"minor_comments":[{"comment":"Equation (2) is not syntactically well-defined: the step function is applied to a single similarity value with no second reward component, so the summation over i is unclear. Please restate it using the pairwise comparison shown in Algorithm 2 in Appendix C.1.","section":"Section 4.2, Eq. (2)"},{"comment":"The 'Txet2Reward' label in Table 1 is a typo, and the aggregated numbers in Table 1 do not match the per-task averages in Appendix E for URDP on Isaac (e.g., average NOE is 349/9 ≈ 38.78, not 39.501; average NLC is 22/9 ≈ 2.444, not 2.495). Please verify all aggregate values.","section":"Table 1 and Appendix E"},{"comment":"The abstract uses 'Bidexterous Manipulation' while the body consistently uses 'Dexterity'; please unify the terminology.","section":"Abstract and Section 2"},{"comment":"The text says 'We examine fore metrics' and then lists four metrics; please fix the typo to 'four'.","section":"Section 5.2"},{"comment":"The text says URDP performed best in 92% of tasks in Appendix E but says it outperforms compared methods on 89% of tasks in Table 7; please reconcile these percentages and state the tie-breaking rule.","section":"Appendix E and Table 7"},{"comment":"No code release or seed specification is mentioned. Given the paper's reliance on stochastic LLM sampling and RL training, releasing code and seeds would substantially aid reproducibility and should be listed in the paper or supplement.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible architecture and a broad evaluation suite, but the current evidence is not yet convincing. The reported numbers are internally inconsistent (Tables 6 vs. 7, the 132% claim), the central uncertainty signal is unvalidated and in tension with Disc-2, and the convergence theorem does not apply to the implemented algorithm. These are fixable with additional experiments and a more careful presentation. I am not recommending rejection because the core idea is reasonable and the issues are addressable within the manuscript's scope, but the authors should be required to provide multi-seed results, a positive control for the uncertainty score, and a resolution of the inconsistency between the uncertainty-based penalty and the demonstrated value of high-uncertainty components."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLM-based reward design. The core insight is solid: LLMs are bad at numeric optimization, so decouple reward-component generation from reward-intensity tuning and hand the tuning to BO. The paper backs this with a nice case study showing Eureka just re-tweaks temperature parameters and oscillates. The deduplication of semantically identical reward samples via BGE-M3 similarity is a practical, sensible efficiency win, and the ablations (Abl-1/2/3) are the right checks. Thirty-five tasks across three benchmarks is real scale, and the per-task tables actually let you see where gains come from.\n\nThe soft spots are real but mostly addressable. First, no seeds, no error bars, single runs—for RL that is not enough to support the headline numbers. Second, the claimed 132% gain over Eureka on Isaac does not match the table: 3.424 vs 1.607 is about 113%. Third, the abstract and conclusion say \"simulation-free identification of ineffective reward components,\" but the method identifies redundant (near-duplicate) components, not ineffective ones. Those are different claims, and the latter is unsupported. Fourth, the U(r) score does double duty: it filters duplicates and it reweights the BO kernel and acquisition. The first use needs no validation beyond \"are these semantically the same?\", which is fine. The second assumes high uncertainty means a dimension should be exploited rather than explored, and the paper never tests that assumption against an oracle or random weighting. The own Disc-2 finding—that high-uncertainty components are often novel and useful—sits awkwardly next to the claim that divergent outputs \"demand more refinement.\"\n\nAppendix F's convergence proof is a limiting-case argument (f constant along d-m directions) that does not connect to the actual uEI weights. It should be reframed as intuition, not a theorem for the proposed algorithm. The limitation section is honest, which counts for something.\n\nThis is a conditional accept, not a reject. Give it to a referee: the framework is new, the scale is respectable, and the flaws are fixable with seeds, error bars, corrected statistics, code release, and a positive control for U(r). I would not cite it as-is, but I would bring it to reading group and watch the revision.","headline":"URDP combines semantic-similarity deduplication with uncertainty-weighted BO for LLM reward design; the decoupling idea is sound and the experiments are extensive, but the evidence is under-powered and the central uncertainty metric is unvalidated.","tokens_in":27981,"tokens_out":2652,"would_cite":false,"duration_ms":35559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that splitting reward design into LLM-proposed components and Bayesian-tuned intensities, with a self-consistency uncertainty score filtering candidates before simulation, yields better policies with half the search…","keywords":["reinforcement learning","reward design","large language models","Bayesian optimization","uncertainty quantification","self-consistency","reward shaping","bi-level optimization"],"falsifier":"Rerun URDP's 35-task pipeline with each component's uncertainty score replaced by random scores drawn from the same distribution while keeping every other step identical; if success rates and simulation counts remain roughly unchanged, the reported gains do not come from the uncertainty information itself, because all that remains is filtering and the UABO penalty shaped by noise.","tokens_in":26862,"feed_emoji":"🎯","tokens_out":7582,"duration_ms":82376,"temperature":0.7,"pith_summary":"This paper is trying to establish that automated reward-function design for reinforcement learning improves when the two tasks inside it are decoupled: deciding what to reward, which suits a large language model, and setting how strongly each reward term counts, which suits a numerical optimizer. The proposed framework, URDP, samples reward-code candidates from an LLM, scores each component by a self-consistency uncertainty measure based on textual and semantic similarity, discards redundant samples before any simulation, and tunes reward intensities with an uncertainty-aware Bayesian optimizer. Across 35 tasks in Isaac, Dexterity, and ManiSkill2, URDP reports higher success rates and human-normalized scores than Eureka and Text2Reward while using about 52% of the simulation episodes and 47% of the LLM calls. The reason a careful reader would care is that, if true, the pipeline turns LLM reward design from an expensive evolutionary search into a cheap alternating optimization, and it identifies a clean division of labor between language-model reasoning and numerical optimization.","feed_headline":"Uncertainty screening halves the cost of robot reward design","feed_subtitle":"LLM-proposed rewards, pre-filtered by uncertainty and tuned by Bayesian optimization, beat baseline search on 35 tasks.","key_machinery":"The load-bearing object is the self-consistency uncertainty score U(ri) defined in Eq. (2): for each LLM-generated reward component, it measures how frequently a near-duplicate appears among the K sampled candidates, using the maximum of a textual similarity (Python SequenceMatcher) and a semantic similarity (BGE-M3 embeddings) with a step threshold ω=0.95; high U means the component is rare or diverse across samples. This score does two jobs. In the outer loop it drives reward-code sampling and filtering, giving high-uncertainty components extra inner-loop refinement because they may be novel. In the inner loop it reshapes the Gaussian-process surrogate: the uncertainty values become dimension-specific length scales in an anisotropic Matern kernel (Eq. 5) and a penalty term in the uncertainty-accelerated expected-improvement acquisition function (Eqs. 7–8), so the optimizer is discouraged from wandering along dimensions with high U and focuses exploitation near the current best point. The surrounding mechanism is the alternating bi-level optimization that separates reward-component reasoning from reward-intensity tuning.","core_discovery":"On the paper's own terms, the central discovery is that the variability of an LLM's own reward-code proposals is usable signal. URDP computes an uncertainty score U(r) for each reward component from how often semantically or textually similar components appear across sampled candidates, treats low-uncertainty components as well-internalized knowledge and high-uncertainty components as novel or under-specified, filters redundant reward functions before simulation, and feeds the same uncertainty values into an anisotropic kernel and a penalized expected-improvement acquisition function so that Bayesian optimization spends fewer evaluations on directions the LLM considers uncertain. The paper claims that this alternating bi-level architecture—LLM in the outer loop for reward logic, UABO in the inner loop for reward intensity—outperforms existing methods in both reward quality and design efficiency: under identical simulation budgets, agents trained on URDP rewards reach higher success rates than Eureka on all three benchmarks, and URDP reaches its peak with 52.4% of Eureka's simulation episodes and 46.6% of its LLM calls; the paper also reports that URDP-designed rewards beat human-designed rewards on 89% of the 35 tasks.","pith_inferences":["Generalizing the same trick: any expensive black-box evaluation of LLM proposals, such as prompt templates, controller gains, or code patches, could be made cheaper by scoring candidate diversity before simulation and using that score to shape the surrogate model.","The novelty-exploration effect could be isolated by holding the UABO kernel fixed and assigning extra inner-loop iterations to the highest-uncertainty components versus randomly selected components, separating real novelty-seeking from mere budget redistribution.","Because uncertainty is computed from code-string similarity, a component that is linguistically novel but functionally identical to existing components would be misclassified as high-uncertainty; a counterfactual test removing such components would reveal how much of the reported novelty effect is semantic versus lexical.","The paper's own limitation note about spatial reasoning suggests the natural stress test: for tasks where grasp paths are blocked by obstacles, similarity-based uncertainty will not see the missing visuospatial constraint, so a video-language-model-grounded variant is the clear next boundary case."],"forward_implications":["If URDP's central claim is right, LLM-based reward design need not spend simulation budget on every sampled candidate; pre-simulation uncertainty screening alone removes redundant evaluations.","A decoupled setup in which Bayesian optimization, not the LLM, tunes reward intensities would become the default design for future automated reward engineering, since the LLM-only evolutionary baseline showed performance regressions and oscillatory behavior.","High-uncertainty reward components, which the framework treats as potentially novel, can act as effective reward-shaping terms; removing them degrades final policy performance.","Comparing URDP with standard BO shows that the uncertainty-aware kernel and acquisition penalty reach comparable reward quality with about 80% of the sampling budget on Isaac tasks.","Across the three benchmarks, URDP outperforms human-designed rewards on 89% of tasks, suggesting automated pipelines can replace manual reward engineering."],"supporting_citations":[{"why":"Eureka is the main evolutionary-search baseline URDP compares against and the source of the Isaac and Dexterity task setups and human reward data.","marker":"Ma et al. (2024a)"},{"why":"Supplies the self-consistency principle that grounds the uncertainty score: consistent LLM outputs are treated as reliable knowledge.","marker":"Wang et al. (2022)"},{"why":"Establishes practical Bayesian optimization, the inner-loop machinery URDP extends.","marker":"Snoek et al. (2012)"},{"why":"Provides the expected-improvement acquisition function that UABO modifies with an uncertainty penalty.","marker":"Ament et al. (2023)"},{"why":"Motivates semantic, rather than surface-level, uncertainty in LLM outputs, matching URDP's semantic-similarity component.","marker":"Kuhn et al. (2023)"},{"why":"BGE-M3 is the embedding model URDP uses to compute the semantic similarity in Eq. (2).","marker":"Xiao et al. (2024)"},{"why":"Supplies the convergence-rate analysis that UABO's proof adapts to lower effective dimensionality.","marker":"Bull (2011)"},{"why":"Text2Reward is the other main LLM reward-design baseline, providing dense reward generation and comparison numbers.","marker":"Xie et al. (2024)"},{"why":"Defines the Dexterity bimanual manipulation benchmark tasks used in evaluation.","marker":"Chen et al. (2022)"},{"why":"DeepSeek-v3 is the LLM backbone used in all experiments.","marker":"Liu et al. (2024)"}],"fun_headline_variants":["Uncertainty scores from LLMs halve simulation cost in reward design","Bi-level LLM-Bayes reward design beats human rewards on 89% of tasks","Uncertainty-aware reward design outperforms Eureka in speed and quality","LLM uncertainty pre-filter cuts reward design cost nearly in half","Uncertainty from LLM self-consistency informs efficient reward optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that how much an LLM's proposed reward components resemble one another in wording and meaning is a reliable signal for whether those components are trustworthy, redundant, or worth exploring.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty scores from LLMs halve simulation cost in reward design","Bi-level LLM-Bayes reward design beats human rewards on 89% of tasks","Uncertainty-aware reward design outperforms Eureka in speed and quality","LLM uncertainty pre-filter cuts reward design cost nearly in half","Uncertainty from LLM self-consistency informs efficient reward optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001295,"raw_usage":{"total_tokens":5325,"prompt_tokens":1027,"completion_tokens":4298,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":4200}},"tokens_in":643,"tokens_out":4298,"duration_ms":30062,"temperature":1.0,"reasoning_tokens":4200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:33:55.069408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun URDP's 35-task pipeline with each component's uncertainty score replaced by random scores drawn from the same distribution while keeping every other step identical; if success rates and simulation counts remain roughly unchanged, the reported gains do not come from the uncertainty information itself, because all that remains is filtering and the UABO penalty shaped by noise.","supporting_citations":[],"review_version":1}