{"id":"bc3561bd-25d7-4532-8c64-882ae5a2f963","arxiv_id":"2412.02089","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Introduces SOBBO, plus an estimate-then-differentiate surrogate method and a conservative-field gradient interpolation method, and shows the latter is more accurate and robust in scarce-data experiments.","lead":"This paper proposes a new optimization setup, Stochastic Offline BBO, where the goal is to optimize a black-box function in expectation over random conditions using only historical data, and introduces two methods for large-data and scarce-data regimes. The work matters because many real-world design problems, from drug discovery to network design, cannot afford active queries and must account for random conditions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ETD consistency theorem (Thm. 3.1) is not proven: the gradient-class GC assumption is unsupported and the proof derives L1 gradient convergence from L1 value convergence, which is false.","rationale":"The central claim has two parts: an asymptotic consistency guarantee for ETD and an empirical claim that DGI is stable and accurate in scarce-data regimes. The theoretical guarantee is the part that can be checked a priori, and here the proof is not sound. The reader's weakest assumption identifies the GC-gradient step; I agree that is a real gap, but the deeper problem is the finite-difference step in Appendix A, which tries to infer derivative convergence from value convergence. That inference is false even under uniform boundedness of the finite differences, so fixing the GC assumption alone would not repair Theorem 3.1. I do not see an internal inconsistency in DGI's formulation; Lemma 4.1 is valid under its assumptions, and the empirical ablations support the claim that conservative-field constraints stabilize gradient estimates. The experiments would be stronger with error bars and additional baselines, but that is not the load-bearing concern here. The appropriate verdict remains CONDITIONAL: the paper's practical contribution may survive, but the stated theorem needs repair or additional hypotheses before the large-data guarantee can be accepted.","tokens_in":19851,"tokens_out":12376,"duration_ms":130303,"concrete_test":"Re-derive Theorem 3.1 step by step. (a) Test the GC step: for G={n^{-1}sin(nθ)} on [0,1], G is a bounded C¹ class but GΘ={cos(nθ)} is not Glivenko–Cantelli, contradicting the assertion in Appendix A. (b) Test the derivative step: instantiate the proof's finite-difference bound with ν_n−ν=sin(nθ)/n and uniform θ; the proof's inequalities would force Eθ|cos(nθ)|→0, which is false (≈2/π). Either failure suffices to show Theorem 3.1 is unproven as stated; if the authors supply a repaired proof under explicit assumptions, the conditional verdict stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest formal claim is Theorem 3.1: Eθ|η(θ)−ηhat_n(θ)|→P 0 for the ETD gradient estimator. The appendix proof breaks in two places. First, it asserts that the gradient class GΘ={∇θ g(θ,·): g∈G, θ∈Θ} is Glivenko–Cantelli because G is continuously differentiable and Θ,X are compact. This does not follow: C¹ boundedness controls pointwise values, not the metric entropy of derivatives; the family of bounded continuous functions is not GC, and GC is not preserved under differentiation. A uniform bound L on ∇θg is also used in the proof but is absent from the theorem statement. Second, even granting the GC step, the first term in Eq. (13) is handled by a finite-difference/Moore–Osgood argument that purports to obtain Eθ|∇θ(ν_n−ν)|→0 from Eθ|ν_n−ν|→0. That implication is false: with ν_n−ν = sin(nθ)/n one has ν_n→ν pointwise and in L1, yet Eθ|∇(ν_n−ν)| = Eθ|cos(nθ)| does not tend to zero. The double-limit interchange fails because convergence in h is not uniform in n. Thus Theorem 3.1 is not established as stated; the large-data consistency guarantee requires either a new proof or materially stronger hypotheses, such as a uniformly bounded and equicontinuous gradient class with derivative control of the ERM excess risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Stochastic Offline Black-Box Optimization (SOBBO), where the expected value ν(θ)=E_X[g(θ,X)] of an expensive black-box function must be optimized using only historical tuples (θ,x,y) with y=g(θ,x)+ε. Two methods are proposed: ETD (Estimate-Then-Differentiate), which fits a differentiable surrogate to the data and uses its θ-gradient averaged over observed x's, and DGI (Deep Gradient Interpolation), which directly trains a network h_φ to approximate ∇_{θ,x}g under conservative-field constraints enforced through balance equations, path-integral reconstruction, and path-independence losses. The paper claims an asymptotic consistency guarantee for ETD (Theorem 3.1) and demonstrates empirically that DGI produces more accurate and stable gradient estimates than ETD in scarce-data regimes on synthetic and simulator-based real-world tasks.","tokens_in":20284,"tokens_out":7415,"duration_ms":66808,"significance":"If the theoretical guarantee for ETD were established and the empirical evaluation properly supported, the paper would make a useful contribution by formalizing a stochastic variant of offline BBO and providing a practical gradient-estimation pipeline that reuses standard gradient-based optimizers. The paper is also commendable for evaluating on simulator-based real-world tasks (SimOpt), for including ablations of the DGI losses, and for reporting noise-robustness experiments. However, the validity of the central consistency theorem is currently not established, and the quantitative evaluation has several reporting gaps; the contribution is therefore conditional.","major_comments":[{"comment":"The proof of Theorem 3.1 is not valid as written, so the claimed consistency of the ETD gradient estimator is not established. Two load-bearing steps fail. First, the proof asserts that GΘ = {∇_θ g(θ,·) : g∈G, θ∈Θ} is Glivenko–Cantelli because G is continuously differentiable and Θ,X are compact; this does not follow from boundedness/continuity of g, and the family of bounded continuous functions is not Glivenko–Cantelli without additional metric-entropy or variation assumptions. The proof also uses a uniform bound L on ∇_θ g that is not stated in the theorem. Second, the Moore–Osgood/finite-difference argument in Eq. (13)–(14) derives E_θ|∇_θ(ν_n−ν)|→0 from E_θ|ν_n−ν|→0; this implication is false, e.g. with ν_n−ν = sin(nθ)/n, pointwise and L1 convergence hold but E_θ|cos(nθ)| does not vanish. The double limit in h and n is interchanged without uniformity in n. Consequently the main large-data guarantee of the paper is unsupported; a corrected proof would need materially stronger assumptions (e.g., uniform boundedness and equicontinuity of the gradient class, plus derivative control of the ERM excess risk) or a different argument.","section":"Theorem 3.1 / Appendix A"},{"comment":"The cosine-similarity metric is mis-specified. The paper defines Cos Sim(ψ, ψ̂) = ||ψ−ψ̂||₂² / (||ψ||₂||ψ̂||₂), which is a normalized squared error, not the cosine similarity between ψ and ψ̂; the standard cosine similarity is (ψ·ψ̂)/(||ψ||₂||ψ̂||₂). The two quantities are not monotonically related unless ||ψ||₂=||ψ̂||₂, so the figures labeled 'cosine similarity (higher is better)' do not report what they claim. This affects the central gradient-accuracy comparison in Figures 3, 4, and the associated text; the metric should be corrected and the experiments re-evaluated.","section":"Section 5.2, Metrics"},{"comment":"The optimization-performance claims are not statistically supported. Although the scarce-data experiments are repeated 50 times and large-data experiments 20 times, Tables 1 and 2 report only averaged values with no standard deviations, confidence intervals, or significance tests, so statements such as 'significantly outperform the baseline of random search' are not justified. Moreover, the baseline set is limited to random search and the (unobtainable) dataset oracle; no offline BBO method from the related literature (e.g., the conservative objective models cited as Trabucco et al., 2021, or other model-based optimizers mentioned in Section 2) is compared. Since the paper claims to advance offline BBO, the absence of such baselines makes it difficult to assess the practical contribution.","section":"Section 5, Tables 1 and 2, and Baselines"}],"minor_comments":[{"comment":"There are typos in the introduction: 'block-box' should be 'black-box', and 'to identity argmin' should be 'to identify argmin'.","section":"Section 1"},{"comment":"In the statement of Lemma 4.1, 'Var[ϵ2] = σ2' should be 'E[ϵ²] = σ²' (or 'Var[ϵ] = σ²'), and the integrand in the displayed conclusion should be h_φ, not ∇_{θ,x}h_φ, to match the proof and the reconstruction principle in Eq. (8).","section":"Lemma 4.1"},{"comment":"The notation h[θ]_φn(x_i, φ) in Eq. (11) is inconsistent; the dependence should be written as h_φn(θ, x_i) (or h[θ]_φn(θ, x_i)) with the parameter argument indicated clearly.","section":"Equation (11)"},{"comment":"The sentence 'this value cannot be achieved in practice because the true value ν(θ) cannot be observed' is confusing, since OC is computed from the true ν rather than from the observed y; the text should clarify that OC is an oracle baseline.","section":"Section 5.1, Oracle in the Dataset"},{"comment":"The claim that the proposed methods 'outperform the (unobtainable) dataset oracle in most scenarios' is not supported by Tables 1 and 2: for example, Table 2 VL shows DGI values of 218 and 221 versus OC 0.0, and Table 1 Zak shows DGI 0.0745 versus OC 0.0248.","section":"Section 5.1, final paragraph"},{"comment":"Minor proof-reading issues: 'Jansen's inequality' should be 'Jensen's inequality'; the estimator is written bgn in the theorem statement but ĝ_n elsewhere; these notations should be unified.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main theorem's proof appears to rely on an incorrect attribution of Glivenko–Cantelli properties to gradient classes of bounded C¹ functions, and the manuscript does not yet provide code for the experiments, which would aid reproducibility. The paper would benefit from a careful re-derivation of the consistency result and a re-run of the gradient-metric evaluation after fixing the cosine formula."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SOBBO is a real problem worth naming, and DGI's conservative-field losses are a creative, well-motivated response to scarce-data offline optimization. The central claim that ETD is asymptotically consistent is not established — the appendix proof has two load-bearing gaps — and the experimental section undercuts its own promise by omitting error bars and offline BBO baselines. I'd still send this to review; the core idea survives the bad theorem.\n\nWhat's new: the SOBBO formulation — optimize ν(θ)=E_X[g(θ,X)] from historical data without active queries — is a natural and useful extension of offline BBO. ETD is surrogate-then-differentiate, and the paper says as much. DGI is the real contribution: directly regressing gradients subject to balance equations, path reconstruction, and path-independence penalties is a sensible inductive bias, and the ablations show each piece helps. The noise-robustness result in Appendix F is also a point in its favor.\n\nSoft spots, in order of severity. First, Theorem 3.1 is not proven as stated. The appendix asserts the gradient class is Glivenko-Cantelli because G is C^1 and Θ, X are compact; that doesn't follow — C^1 boundedness does not control the metric entropy of derivatives, and the bounded-gradient bound L appears in the proof but not in the theorem statement. More seriously, the proof derives L1 convergence of gradients from L1 convergence of values via a finite-difference/Moore–Osgood argument. That implication is false: ν_n−ν = sin(nθ)/n goes to 0 in L1 while E|cos(nθ)| does not. So the asymptotic guarantee for ETD is not established. The authors need stronger hypotheses (e.g., uniform equicontinuity of the gradient class plus control of the ERM excess risk) or should label the consistency claim a heuristic. Second, Tables 1 and 2 have no error bars or significance, despite the paper saying experiments were repeated 50 and 20 times. Fixable but mandatory. Third, the comparisons are random search and the dataset oracle; no offline BBO baseline (COMs, etc.) appears, so 'outperforms' is weaker than claimed. Fourth, the cosine-similarity formula in Section 5.2 is written as a squared distance yet called cosine similarity with 'higher is better'; that needs fixing. None of this sinks DGI's empirical story, but the theoretical headline should be rewritten.\n\nWho it's for: people working on offline optimization under uncertainty; they should read the DGI sections and treat Theorem 3.1 as open. A serious referee should engage, because the problem framing and DGI are worth developing. My recommendation: send to peer review, with a request to correct the theorem or downgrade it to a heuristic, add error bars, and include at least one offline BBO baseline.","headline":"A genuinely new problem framing and a creative DGI loss, but the ETD consistency theorem is not proven and the experiments need error bars and real baselines.","tokens_in":20687,"tokens_out":4214,"would_cite":true,"duration_ms":40607,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C15","68T07","62G05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline data alone can guide stochastic black-box optimization","keywords":["offline black-box optimization","stochastic optimization","gradient estimation","conservative field","path integral","Glivenko-Cantelli class","deep gradient interpolation","data efficiency"],"falsifier":"Choose a compact domain and a bounded, continuously differentiable function class that is known not to be Glivenko-Cantelli, train an ETD surrogate by ERM on an offline dataset with increasing $n$, and measure $E_\\theta\\|\\eta(\\theta)-\\hat\\eta_n(\\theta)\\|$; if it stays away from zero as $n$ grows, the consistency claim fails as stated without the missing boundedness assumption. A second direct check is to reproduce the noise ablation of Figure 10 and see whether DGI's gradient cosine similarity stays above 0.9 while ETD's drops below 0.7 at single-sample SNR of 1.","tokens_in":1910,"feed_emoji":"🎯","tokens_out":3022,"duration_ms":85849,"temperature":0.7,"pith_summary":"This paper introduces Stochastic Offline Black-Box Optimization (SOBBO), the task of minimizing an expected objective $E_X[g(\\theta,X)]$ when the function $g$ is a black box and only a historical dataset of $(\\theta,x,y)$ samples is available. The authors propose two gradient-recovery strategies: Estimate-Then-Differentiate (ETD), which fits a differentiable surrogate and differentiates it, and Deep Gradient Interpolation (DGI), which learns the gradient function directly while enforcing the conservative-field structure that any true gradient must satisfy. They prove that the ETD gradient estimate converges in probability to the true gradient under Glivenko-Cantelli conditions, and they show empirically that both methods beat random search and often the best design in the dataset. If the claims hold, expensive stochastic black-box systems can be optimized with existing data, without online evaluation.","feed_headline":"Offline data alone can guide stochastic black-box optimization","feed_subtitle":"Two estimators recover objective gradients from historical samples, beating random search on scarce data.","key_machinery":"Two estimation pipelines carry the argument. ETD works through surrogate consistency: ERM on a Glivenko-Cantelli class yields $L^2$ convergence of the fitted function, and differentiating that function transfers the convergence to the gradient estimator. DGI works through the calculus structure of gradients: the balance equations from Clairaut's theorem force the learned field $h_\\phi$ to be conservative, and path-integral reconstruction from the fundamental theorem of calculus forces it to reproduce observed objective differences. The reconstruction loss $L_r$ has minimum $2\\sigma^2$ and pins the gradient to the true one in the noise-free sense, while the worst-case path loss $L_e$ over randomly sampled polynomial paths makes the integral path-independent. These constraints are the difference between DGI and ETD in scarce-data experiments, and the ablations show that removing them makes DGI behave like ETD.","core_discovery":"The central claim is that the gradient of a stochastic black-box objective can be recovered from offline samples well enough to support gradient-based optimization. In the large-data regime, ETD fits a differentiable surrogate $g_\\phi$ to the noisy observations by empirical risk minimization and estimates the gradient of the value function $\\nu(\\theta)=E_X[g(\\theta,X)]$ as $\\widehat{\\nabla\\nu}_n(\\theta)=\\frac{1}{n}\\sum_{k=1}^n \\nabla_\\theta g_{\\hat\\phi_n}(\\theta,x_k)$. Theorem 3.1 states that this estimate converges in probability to the true gradient on average, i.e. $E_\\theta\\|\\eta(\\theta)-\\hat\\eta_n(\\theta)\\|\\xrightarrow{P}0$, provided the function class is Glivenko-Cantelli. In the scarce-data regime, DGI trains $h_\\phi\\approx\\nabla_{\\theta,x}g$ directly with a path-integral reconstruction loss plus a balance loss enforcing $\\partial h^j_\\phi/\\partial\\zeta^i=\\partial h^i_\\phi/\\partial\\zeta^j$, so that the fitted vector field is conservative. The reported experiments show DGI keeps accurate gradient estimates on datasets of 128 samples, where ETD's estimates degrade as training overfits the label noise, while ETD remains competitive on larger datasets.","pith_inferences":["An extension the authors leave implicit: the same conservative-field estimator could be applied to offline reinforcement learning, where logged transitions are historical samples of an expected-return function and the policy gradient is the quantity to recover.","The consistency proof assumes bounded gradients without stating that assumption in the theorem; applying ETD to neural networks in practice would benefit from checking whether the fitted class satisfies the Glivenko-Cantelli condition, since overly expressive surrogates might break the guarantee.","DGI's path-integral losses could be adapted to enforce symmetry conditions for higher-order derivatives, such as Hessians or Jacobians, giving second-order offline optimization methods the same historical-data advantage.","The noise-ablation result suggests a testable use beyond the paper: DGI could serve as a denoising gradient estimator in online settings with cheap but noisy simulators."],"forward_implications":["With a surrogate that converges in $L^2$, differentiating it yields a consistent estimator of the expected-objective gradient, so standard gradient-based optimizers such as SGD, Adam, and momentum can be applied to offline stochastic optimization problems.","Gradient-based optimization on historical data alone can improve over random search and often over the best observed design in the offline dataset, including on stochastic queue, newsvendor, and project-network problems.","Enforcing the conservative-field and path-independence constraints is what separates DGI from ETD in the scarce-data regime; without them, DGI's gradient estimation degrades to ETD's level.","DGI maintains higher gradient cosine similarity than ETD under strong label noise, so its advantage grows as the observation noise increases.","The consistency result does not require sample splitting: the stored $x$ values used to fit the surrogate can also be reused to estimate gradients."],"supporting_citations":[{"why":"Defines offline black-box optimization and motivates optimizing from historical data, the problem setting this paper extends to stochastic objectives.","marker":"Trabucco et al., 2021"},{"why":"Provides the sample average approximation template on which the ETD gradient estimator is built.","marker":"Kim et al., 2015"},{"why":"Supplies the empirical process machinery used in the proof of Theorem 3.1 to get $L^2$ convergence of the ERM surrogate.","marker":"Shorack and Wellner, 2009"},{"why":"Establishes the risk-minimization principles behind the Glivenko-Cantelli argument in the consistency proof.","marker":"Vapnik, 1991"},{"why":"Provides the strong law of large numbers that motivates the sample-average gradient estimator before the black-box assumption is introduced.","marker":"Renyi, 2007"},{"why":"Supplies the SimOpt simulators used for the real-world newsvendor, M/M/1 queue, and stochastic activity network experiments.","marker":"Eckman et al., 2023"}],"fun_headline_variants":["Offline gradients beat random search in stochastic BBO","Stochastic BBO from offline samples: two gradient estimators","Gradient-based offline BBO for stochastic objectives","SOBBO: recover gradients from offline data for stochastic BBO"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The paper's guarantee collapses if the fitted function class is too expressive or its gradients are unbounded; the consistency proof assumes a Glivenko-Cantelli class and bounded gradients.","fun_headline_variants_meta":{"raw":{"variants":["Offline gradients beat random search in stochastic BBO","Stochastic BBO from offline samples: two gradient estimators","Gradient-based offline BBO for stochastic objectives","SOBBO: recover gradients from offline data for stochastic BBO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2864,"prompt_tokens":949,"completion_tokens":1915,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":565,"tokens_out":1915,"duration_ms":12833,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:50:36.549710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Choose a compact domain and a bounded, continuously differentiable function class that is known not to be Glivenko-Cantelli, train an ETD surrogate by ERM on an offline dataset with increasing $n$, and measure $E_\\theta\\|\\eta(\\theta)-\\hat\\eta_n(\\theta)\\|$; if it stays away from zero as $n$ grows, the consistency claim fails as stated without the missing boundedness assumption. A second direct check is to reproduce the noise ablation of Figure 10 and see whether DGI's gradient cosine similarity stays above 0.9 while ETD's drops below 0.7 at single-sample SNR of 1.","supporting_citations":[],"review_version":1}