{"id":"57e484b2-ff7e-4709-a681-d4139bc331f7","arxiv_id":"2412.11051","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A generative optimization method that jointly samples discrete and continuous design variables outperforms decoupled skeleton-then-optimize baselines in sample efficiency across bitstring, decision-tree, and symbolic regression tasks.","lead":"This paper introduces DisCo-DSO, a reinforcement-learning method that generates both the discrete structure and the continuous parameters of a design in one step, so each candidate costs one objective evaluation. It reports gains in sample efficiency over decoupled methods on decision-tree policies for control, symbolic regression, and a bitstring toy problem.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Decision-tree baselines optimize thresholds without the prefix-dependent bounds that DisCo-DSO uses, confounding the claimed advantage.","rationale":"The reader's weakest assumption targets the expressiveness of D (fixed sigma=0.5). That is a plausible exploration-efficiency worry, but it is not a hard representational limit: the truncated normal's support is exactly the feasible interval, and the normal used for SR constants has unbounded support; the model can shift its mean. In the bitstring task, DisCo-DSO's margin is largest when continuous parameters matter most (alpha=0.5, Table 3), which is the opposite of what a severe expressiveness failure would predict. The baseline-fairness issue I identify is different and more direct. In the decision-tree experiments, DisCo-DSO receives prefix-dependent bounds via Algorithm 3; the decoupled baselines are described as using unconstrained SciPy optimizers on the continuous thresholds. If those solvers propose thresholds that violate the tree's prefix ordering, the resulting trees can be semantically invalid, making the baselines artificially weak. This is not a disagreement with consensus; it is an internal asymmetry in the evaluation protocol. The paper does not state that the baselines enforce the same bounds, and the appendix's constraint discussion is framed only for autoregressive sampling. The parameterized bitstring and symbolic regression tasks lack such constraints, so they cannot serve as controls for the decision-tree confound. Consequently, the headline claim that DisCo-DSO outperforms state-of-the-art decision-tree policy methods is conditionally supported at best. I recommend keeping the CONDITIONAL verdict until the authors either constrain the decoupled baselines or show that the fraction of infeasible threshold proposals is negligible.","tokens_in":32802,"tokens_out":10285,"duration_ms":93230,"concrete_test":"For LunarLander-v2 and Acrobot-v1, rerun Decoupled-RL-BFGS and Decoupled-GP-BFGS with the downstream solver's proposed thresholds projected onto the Algorithm 3 bounds at each evaluation (e.g., clamp left-child thresholds to [beta_min, beta_p - h] and right-child thresholds to [beta_p + h, beta_max] for the relevant feature). Compare the best-reward-versus-episodes curves to the original results. Also compute the fraction of original unconstrained solver proposals that violate these bounds. If the gap to DisCo-DSO collapses below one standard error, or if a large fraction of original proposals are infeasible, the superiority claim on decision trees is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.2 (Decision tree policies), DisCo-DSO samples decision-tree thresholds from truncated normals whose intervals are propagated by Algorithm 3: a left child of x_n < beta_p gets upper bound beta_p - h, a right child gets lower bound beta_p + h. The decoupled baselines (Decoupled-RL-{BFGS,anneal,evo}, Decoupled-GP-{BFGS,anneal,evo}) are described as optimizing continuous parameters with SciPy solvers, and no corresponding constraint handling is specified anywhere in the paper. An unconstrained solver can propose a left-child threshold above the parent threshold on the same feature, yielding a tree with an unreachable branch or semantics different from the skeleton the generative model scored. Thus the decoupled baselines search over a superset of semantically valid trees and are systematically handicapped. The parameterized-bitstring and symbolic-regression tasks have no prefix-dependent constraints, so they cannot control for this confound in the decision-tree experiments. Yet those experiments support the abstract's headline 'state-of-the-art' claim for interpretable RL with decision trees. The apparent sample-efficiency advantage of joint sampling over decoupled optimization on decision trees could therefore be an artifact of comparing constrained joint sampling to unconstrained decoupled optimization.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DisCo-DSO, an autoregressive generative model that jointly samples discrete tokens and their continuous parameters for black-box optimization in hybrid, variable-length spaces. The model is trained with a risk-seeking policy gradient extended to mixed discrete-continuous distributions, with continuous parameters drawn from a normal or truncated-normal distribution whose location is emitted by the model. The method is evaluated on a parameterized bitstring benchmark, decision-tree policy search in four RL environments, and a 45-benchmark symbolic regression suite, and is compared against decoupled RL/GP baselines and several literature methods. The central claim is that joint generative sampling is more sample-efficient and yields better designs than skeleton-then-optimize approaches, with the advantage growing as problem complexity increases.","tokens_in":33083,"tokens_out":23257,"duration_ms":201792,"significance":"If the empirical claims hold, DisCo-DSO would make joint generative sampling a strong default for hybrid black-box optimization and would extend deep symbolic optimization in a useful direction. The risk-seeking gradient derivation in the appendix is correct, and the bitstring and symbolic-regression experiments are consistent with the claimed sample-efficiency mechanism. The paper is also honest about several limitations, including reliance on domain-specific ranges and the restriction to univariate decision trees. However, the decision-tree experiments contain a constraint-handling confound, the bound-propagation algorithm is under-specified and internally inconsistent, and the fixed continuous scale in Table 12 is a restrictive design choice that is not tested. These issues need to be resolved before the headline claims are fully supported.","major_comments":[{"comment":"The decision-tree experiments confound joint sampling with constraint handling. DisCo-DSO samples each threshold from a truncated normal whose bounds are propagated by Algorithm 3, so every sampled tree respects parent-child ordering on thresholds. The Decoupled-RL-* and Decoupled-GP-* baselines are described only as optimizing continuous parameters with SciPy solvers, and no prefix-dependent bound constraints are specified for them. An unconstrained solver can set a left-child threshold above its parent threshold on the same feature, producing an unreachable branch and a tree whose semantics differ from the skeleton that the generative model scored. Those baselines therefore search over a superset of semantically valid trees and are systematically handicapped. Because the bitstring and symbolic-regression tasks have no prefix-dependent constraints, they cannot control for this confound. Please either apply the same bound propagation to the decoupled baselines or report a decoupled variant that is constrained consistently, and state the outcome.","section":"§4.2 (Decision tree policies) and Algorithm 3"},{"comment":"Algorithm 3 is under-specified and internally inconsistent. The resolution h > 0 is never assigned a value in Table 12 or anywhere else, so the decision-tree results cannot be reproduced from the text. More seriously, the fallback rule that maintains a minimal distance can return a range that violates the parent threshold. For example, with parent bounds (0, 5), h = 0.5, and parent threshold beta_p = 0.1, the left child gets beta_max = 0.1 - 0.5 = -0.4, which is below beta_min = 0; the rule then sets beta_max = 0 + 0.25 = 0.25, which is larger than beta_p. A left-child threshold in (0.1, 0.25] makes the right branch of that child unreachable, so the sampled object is not a semantically valid decision tree. In addition, the bounds shown for the node x1 < 3 in Figure 3 (beta_1 in (2, 3)) cannot be derived from the parent-only updates in Algorithm 3 when the parent is x2 < 6. Please specify the full propagation over all active ancestor constraints and give the actual value of h used.","section":"Algorithm 3 and Figure 3"},{"comment":"The fixed continuous-parameter distribution is a restrictive design choice whose consequences are not tested. Table 12 fixes sigma = 0.5, disables learning the scale, and the paper does not report the initialization of the location parameter phi emitted by the autoregressive model. For a normal with sigma = 0.5, the probability of drawing a value more than about three units from the current location is negligible, so exploration of the continuous component is effectively local. On symbolic-regression instances where the optimal constants are far from the initial phi, DisCo-DSO would need many gradient steps before it can even sample near the optimum, and if the reward is uninformative away from the optimum the risk-seeking estimator may not provide a directional signal. The paper should report the initialization, run a sensitivity sweep over sigma (or allow a learned scale), and discuss how the method behaves when the optimal constants are far from the initial location. Without this, the symbolic-regression comparison does not yet establish the claimed robustness to non-differentiable objectives in general.","section":"Table 12 and §4.3 (symbolic regression)"}],"minor_comments":[{"comment":"The line \"Release code is LLNL-CONF-854776\" is a release identifier, not a URL or repository; for an empirical machine-learning paper, please provide a persistent link to the code and data used in the experiments.","section":"Acknowledgments"},{"comment":"The headers of Tables 8 and 9 contain \"All Table 3\", which appears to be a copy-paste artifact and should be corrected to \"All\".","section":"Tables 8 and 9"},{"comment":"In Table 4, the CartPole-v1 gaps for Decoupled-RL-evo and Decoupled-RL-anneal are 0.00, so the statement that DisCo-DSO outperforms decoupled approaches on all tasks should be qualified as a sample-efficiency advantage rather than a final best-reward advantage on every environment.","section":"Table 4"},{"comment":"The BO baseline is described as performing fewer function evaluations than the other methods because of its O(n^3) complexity, but the x-axis is the number of function evaluations; please clarify whether the comparison is under a wall-clock budget or some other computational budget.","section":"Figure 2"},{"comment":"The x-axis is labeled \"Number of environment episodes\" while the text discusses objective-function evaluations; because each objective evaluation uses N = 100 episodes, please state the conversion in the caption.","section":"Figure 4"},{"comment":"Table 10 lists a \"Moving average coefficient (beta)\" and Algorithm 1 does not appear to use it; please clarify what quantity this coefficient smooths and where it enters the update.","section":"Table 10"},{"comment":"The entry \"Parameter shift 0.0\" is not explained anywhere; please define it and state whether it is applied to the location, the sampled value, or something else.","section":"Table 12"},{"comment":"The sentence describing Hausknecht and Stone [2016] as handling \"continuous state and action spaces\" should say \"parameterized action spaces,\" which is the relevant hybrid-action setting.","section":"Related work"},{"comment":"The abstract's claim of superiority over state-of-the-art methods for interpretable RL with decision trees is scoped to univariate trees only in the conclusion; the abstract should state this scope, especially because oblique decision trees in Custode and Iacca [2023] are not compared.","section":"Abstract and Table 1"},{"comment":"The formal definition of prefix-dependent positional constraints only masks discrete tokens (C subset of L), but the decision-tree parameter bounds in Algorithm 3 constrain the continuous parameter beta_i and are not covered by the formal definition or by the logit prior; please extend the problem statement to represent continuous-parameter constraints.","section":"§3 (Problem definition)"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern about the decision-tree baselines: the constraint-handling confound is the main reason I am recommending major revision rather than acceptance. The paper is otherwise a useful extension of DSO, the policy-gradient derivation is correct, and the experiments are substantial. I would also ask the editor to ensure that the abstract's 'state-of-the-art' claim is scoped to univariate decision trees and to the specific comparisons made in Table 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this is a real extension of the DSO line—jointly sampling discrete and continuous tokens with one autoregressive model trained with risk-seeking policy gradients—and the math is sound. The parameterized-bitstring and symbolic-regression results are credible and support the sample-efficiency story. But the decision-tree experiment has a comparison problem that could inflate the headline advantage, and the paper ships no code. I'd send it out for review, but with a request to fix the baseline before it's published.\n\nThe new piece is genuine. Prior DSO methods sampled a discrete skeleton and then ran a downstream optimizer on the continuous parameters. Here the continuous parameters are drawn from a learned distribution conditioned on the token, and the risk-seeking gradient derivation in the appendix is correct. The bound-propagation routine for decision-tree thresholds (Algorithm 3) is a nice touch, though it's also where the trouble starts.\n\nIn the decision-tree experiments, DisCo-DSO uses prefix-dependent bounds: a child's threshold must respect the parent's threshold on the same feature. The decoupled baselines are described as using off-the-shelf SciPy solvers on the continuous parameters, and there's no indication they're given the same bounds. That means they're searching over a superset of semantically valid trees—left-child thresholds that make branches unreachable, etc. That can handicap them in a way that has nothing to do with joint vs. decoupled sampling. The parameterized-bitstring and symbolic-regression tasks don't have these prefix constraints, so they can't control for this. The decision-tree results are a substantial part of the paper, and the abstract's 'state-of-the-art' claim rests heavily on them. So this is a real soft spot, not a nitpick.\n\nOther issues: the paper excludes oblique trees from Custode and Iacca, which reportedly do better on LunarLander, so the SOTA claim is broader than the univariate scope. And there's no code or data release, which makes independent verification hard. The fixed-scale normal (σ=0.5) for continuous parameters is a limitation, but the paper acknowledges it in the conclusion.\n\nWho's this for? Anyone working on hybrid black-box optimization, symbolic regression, or interpretable RL policies. The core idea is worth taking seriously. I'd accept it for peer review, but I'd insist on a fair comparison for the decision-tree baselines and ideally code release. The central promise—joint sampling saves evaluations—looks real on the tasks without the constraint confound.","headline":"A genuine joint-sampling extension of DSO with sound math, but the decision-tree baseline comparison is confounded by missing constraint handling, and the SOTA claim overreaches.","tokens_in":33545,"tokens_out":3610,"would_cite":true,"duration_ms":34230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that jointly sampling discrete designs and their continuous parameters in one autoregressive pass finds better black-box optima per evaluation than decoupled skeleton-then-optimize approaches, with the gap widening on…","keywords":["discrete-continuous optimization","generative design","autoregressive model","risk-seeking policy gradient","decision tree policies","symbolic regression","sample efficiency","hybrid action space"],"falsifier":"Run DisCo-DSO and a decoupled optimizer on a synthetic hybrid problem where the global optimum is known and its continuous parameter lies at the boundary of the propagated bound or is much narrower than $0.5$; if DisCo-DSO reliably misses it while the decoupled method finds it within the same evaluation budget, the distributional and bound assumptions are the limiting factor.","tokens_in":32593,"feed_emoji":"🎯","tokens_out":6894,"duration_ms":56975,"temperature":0.7,"pith_summary":"The paper tries to establish that a single autoregressive model can learn a joint distribution over discrete tokens and their continuous parameters, so that sampling a complete candidate design costs one objective evaluation instead of a skeleton plus many inner optimization steps. The authors argue that this joint coupling lets the discrete and continuous parts of a design inform each other during search, and that this becomes more valuable as problem complexity grows. They support the claim on a parameterized bitstring benchmark, on decision-tree policies for four reinforcement learning environments, and on a 45-benchmark symbolic regression suite, where joint sampling outperforms the decoupled baselines. A sympathetic reader would take away that for black-box hybrid optimization with variable-length structure and prefix-dependent constraints, joint generative sampling is a better default than skeleton-then-optimize.","feed_headline":"One pass samples structure and numbers, beating two-stage search","feed_subtitle":"A joint autoregressive model turns each sample into a complete design, winning on decision trees and symbolic regression.","key_machinery":"The central object is the joint token distribution $p((l_i, \\beta_i) \\mid (l, \\beta)_{1:i-1}, \\theta)$, with discrete logits $\\psi^{(i)}$ and continuous distribution $D(\\beta \\mid l, \\phi^{(i)})$ emitted per step; for parametrized tokens $D$ is a normal or truncated-normal with fixed scale $\\sigma = 0.5$. The distribution is trained with the risk-seeking policy gradient $J_\\varepsilon(\\theta) = \\mathbb{E}[R(\\tau) \\mid R(\\tau) \\geq R_\\varepsilon(\\theta)] \\log p(\\tau \\mid \\theta)$, which concentrates probability on above-quantile designs. For decision trees, Algorithm 3 propagates parent-threshold bounds down the tree so each threshold is sampled from a truncated normal within its feasible interval, and prefix-dependent constraints mask infeasible tokens in the logits.","core_discovery":"DisCo-DSO extends autoregressive deep symbolic optimization by having the model emit, at each step, both a logit vector over token types and the parameters of a continuous distribution for tokens that carry a numeric value; the token and its parameter are sampled together under a risk-seeking policy gradient that conditions on rewards above a quantile threshold. This makes every sampled sequence a complete design, so the reward can be assigned immediately, and it makes the continuous parameters part of the learned generative process rather than a post-hoc optimization task. The paper claims this joint treatment produces better final rewards per function evaluation on all tested tasks, with the margin over decoupled approaches widening on harder problems.","pith_inferences":["The fixed scale $\\sigma = 0.5$ for the continuous generator is a design choice the paper does not vary; one immediate test is to let the model learn a per-token scale, which could matter for problems whose optimal constants span very different magnitudes.","The bound-propagation rule for decision-tree thresholds is a small recursive procedure that could transfer to any generative model over nested comparisons, such as rule lists or program sketches, wherever a parent condition tightens the feasible range of a child.","If the sample-efficiency result is general, then in expensive black-box settings the first question shifts from 'which downstream optimizer?' to 'can the whole design be sampled jointly?', since the evaluation budget is the real currency.","The paper compares against decoupled methods with fixed inner-optimizer budgets; a decisive stress test would let the decoupled methods spend far more evaluations per skeleton and see whether the joint advantage survives."],"forward_implications":["Every candidate design is complete at sampling time, so the reward from one design costs exactly one objective evaluation; the decoupled baselines spend multiple evaluations to refine a single skeleton.","Because the discrete token and its continuous parameter share one learned distribution, information from past high-reward designs can steer both the structure and the numeric values at once, which should help avoid local optima that a fixed skeleton cannot escape.","On the decision-tree RL benchmarks the learned trees are univariate, compact, and match or beat prior evolutionary, cascading, and differentiable-tree policies at lower or comparable parameter counts.","In symbolic regression the joint approach reaches the best average test-set reward with the fewest function evaluations and avoids the bloat-driven overfitting seen in the genetic-programming baselines.","The reported trend that the advantage grows with complexity implies the method is aimed at settings where evaluation is expensive and hybrid structure is nontrivial."],"supporting_citations":[{"why":"This work establishes the autoregressive-model-plus-reinforcement-learning paradigm for combinatorial optimization that DisCo-DSO extends.","marker":"Bello et al. [2016]"},{"why":"This work supplies the risk-seeking policy gradient and the Deep Symbolic Regression baseline that DisCo-DSO improves on.","marker":"Petersen et al. [2021a]"},{"why":"This work motivates the expensive-evaluation control setting and the decoupled skeleton-then-optimize pipeline DisCo-DSO replaces.","marker":"Landajuela et al. [2021b]"},{"why":"This work introduces parameterized action spaces, the hybrid-action idea behind sampling a token and its continuous parameter together.","marker":"Hausknecht and Stone [2016]"},{"why":"This work provides the evolutionary decision-tree baseline whose reported results DisCo-DSO surpasses.","marker":"Custode and Iacca [2023]"},{"why":"This work provides the interpretable differentiable decision tree baseline compared in the reinforcement learning evaluation.","marker":"Silva et al. [2020]"},{"why":"This work exemplifies the tokenized-constant alternative to continuous sampling in symbolic regression, a key point of contrast.","marker":"Kamienny et al. [2022]"},{"why":"This work defines the genetic-programming symbolic regression baseline and the bloat/overfitting comparison.","marker":"Koza [1994]"},{"why":"This work supplies differential evolution, one of the downstream optimizers used in the decoupled baselines.","marker":"Storn and Price [1997]"}],"fun_headline_variants":["Joint discrete-continuous sampling beats two-stage search","One model samples structure and numbers together","DisCo-DSO couples discrete and continuous optimization","Sample complete hybrid designs in a single pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fixed normal or truncated-normal family with scale $\\sigma = 0.5$, together with the propagated threshold bounds, can represent the optimal continuous parameters; if the optimum lies outside those ranges or needs a different spread, the joint search cannot express it and the claimed advantage would collapse on exactly the hard cases.","fun_headline_variants_meta":{"raw":{"variants":["Joint discrete-continuous sampling beats two-stage search","One model samples structure and numbers together","DisCo-DSO couples discrete and continuous optimization","Sample complete hybrid designs in a single pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1163,"prompt_tokens":838,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":454,"tokens_out":325,"duration_ms":3393,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:21:31.349460+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DisCo-DSO and a decoupled optimizer on a synthetic hybrid problem where the global optimum is known and its continuous parameter lies at the boundary of the propagated bound or is much narrower than $0.5$; if DisCo-DSO reliably misses it while the decoupled method finds it within the same evaluation budget, the distributional and bound assumptions are the limiting factor.","supporting_citations":[],"review_version":1}