{"id":"c5792889-885c-4930-853a-d140adb010a2","arxiv_id":"2506.14460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Randomized finite-difference zeroth-order optimization is exactly single-step REINFORCE with baseline f(theta; xi), and adding an averaged baseline plus query reuse yields a faster ZOO algorithm (ZoAR).","lead":"Zeroth-order optimization with randomized finite differences is shown to be algebraically identical to single-step policy optimization (REINFORCE) with a specific baseline, and the paper uses that view to build a faster optimizer, ZoAR, that averages past function evaluations and reuses old queries.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sphere-sampling bias analysis in Appx B is internally inconsistent: the claimed unbiasedness of ZoAR's gradient estimator (Thm B.3) fails for u~Unif(S^{d-1}), invalidating the variance and convergence guarantees built on it.","rationale":"The headline equivalence in Thm 3.1 and Thm 3.2 is algebraically correct: substituting x=theta+mu u and b(xi)=f(theta;xi) turns the REINFORCE estimator (9) into the ZOO estimator (2). I agree with the reader that this equivalence is largely a re-labeling, and that the genuinely new algorithmic piece is ZoAR. The load-bearing problem I find is not the query-reuse bias floor B_2, although that is also real, but a more basic mathematical error in the theoretical analysis of ZoAR. Appx B states that all proofs are conducted under u~Unif(S^{d-1}), and Thm B.3 claims that ZoAR's estimator (13) is unbiased for the average of historical smoothed gradients. The proof (Appx C.4, step (b)) uses the identity E[F(theta+mu u)u/mu] = nabla F_mu(theta), which holds for Gaussian smoothing but not for uniform sphere sampling. For a linear function the expectation is a/d, not a. The proof's parenthetical redefinition of F_mu as ball smoothing does not repair the argument because the estimator lacks the d/mu factor that Flaxman's lemma requires. Thus Thm B.3, and therefore Thm B.4, Thm B.5, and Thm C.4, do not follow as stated. Since the paper's second and third contributions depend on this appendix, the central algorithmic claim that ZoAR's averaged baseline and query reuse provably reduce variance and improve convergence is unsupported. This is an internal inconsistency, not a disagreement with consensus. A revision would need to redo Appx B with a consistent distribution and the correct scaling, or restrict the theory to Gaussian sampling and rederive the baseline and variance results; the experiments should also state the perturbation distribution and report error bars. As written, the theoretical support for the signature algorithm fails, so I would move the verdict from CONDITIONAL to REJECT, while noting that the Gaussian equivalence result itself remains correct.","tokens_in":35868,"tokens_out":9162,"duration_ms":95122,"concrete_test":"Verify the unbiasedness claim analytically: fix d=2, F(theta)=a·theta, and u~Unif(S^1). Compute E[(F(theta+mu u)-F(theta))/mu u] = E[(a·u)u] = a/2, whereas nabla F_mu(theta)=a. The same computation for any d>=2 gives a/d, not a. If Thm B.3 were correct for sphere sampling, this elementary check would return a; it returns a/d. This one-line test settles whether the Appx B analysis applies to the stated distribution, independently of the B_2 floor and of which perturbation distribution the experiments used.","verdict_should_be":"REJECT","load_bearing_attack":"Appx B proves ZoAR's query-reuse estimator under u~Unif(S^{d-1}), but the central identity used in the proof of Thm B.3 (Appx C.4) is false for that distribution. With F_mu defined in Eq. (3) as E_u[F(theta+mu u)] over the sampling distribution, the estimate E[(F(theta+mu u)-F(theta))/mu u] is not nabla F_mu(theta) when u is uniform on the sphere. For F(theta)=a·theta, E[(F(theta+mu u)-F(theta))/mu u]=E[(a·u)u]=a/d, while nabla F_mu(theta)=a. The cited Flaxman et al. 2005b Lemma 2.1 gives E[F(theta+mu u)u/mu]=(1/d) nabla F_mu^B(theta) for the ball-smoothed objective, so even reinterpreting F_mu as ball smoothing leaves a missing factor d and an objective mismatch. Indeed, Appx C.4 acknowledges this by redefining F_mu as E_{u~Unif(B^d)}[F(theta+mu u)], which is inconsistent with Eq. (3). Consequently Thm B.3 does not hold as stated, and Thm B.4, Thm B.5, and the formal convergence bound Thm C.4 (Eq. 49) inherit the error. The empirical claims may survive, but the paper's theoretical support for ZoAR's variance reduction and convergence is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that Gaussian-smoothed zeroth-order gradient estimation is exactly single-step REINFORCE with a baseline. Section 3 defines a single-step policy objective with policy x=theta+mu u, shows J=F_mu, and shows by substitution that estimator (9) with b(xi)=f(theta;xi) coincides with estimator (2). Section 4 proposes ZoAR, which replaces the single-point baseline with an average of recent function values and reuses historical queries; Appx B contains bias, optimal-baseline, variance, and convergence theorems under u~Unif(S^{d-1}). Experiments on synthetic functions, adversarial attacks, and LLM fine-tuning report faster convergence. The algebraic core Thm 3.2 is correct; however, the Appx B theory has a fundamental sphere-smoothing error, the importance-sampling generalization is not mathematically valid for singular proposals, and the formal convergence bound has a nonvanishing floor. The empirical comparisons may still be valid, but they cannot compensate for the theoretical gaps as they stand.","tokens_in":36140,"tokens_out":18116,"duration_ms":175284,"significance":"The exact equivalence in Thm 3.2 is verifiable and provides a clean vocabulary transfer between ZOO and REINFORCE; this is a useful interpretive contribution, although it is essentially a constructed identity rather than a deep structural theorem. The proposed ZoAR algorithm is simple, and the reported empirical improvements over Vanilla ZOO, ZoHS, and ReLIZO in Figs. 1, 3, Table 1, and Fig. 4 are a genuine strength. However, the second contribution's theoretical justification is load-bearing and the appendix results fail as stated. The paper does not include code, but the experimental protocol with shared update rules and hyperparameters is described in Appx D. In its current form the significance is limited by the invalid Appx B; a thorough revision is needed.","major_comments":[{"comment":"The claim that estimator (13) is unbiased for the average of nabla F_mu(theta_{t-n}) fails for u~Unif(S^{d-1}). For F(theta)=a*theta, E[(F(theta+mu u)-F(theta))/mu u] = a/d, while Eq. (3) with u~Unif(S^{d-1}) defines F_mu with gradient a. The proof's step (b) cites Flaxman et al. (2005b) Lemma 2.1 but redefines F_mu as E_{u~Unif(B^d)}[F(theta+mu u)], which is not the objective in Eq. (3); even for that ball-smoothed objective, the lemma gives E[F(theta+mu u)u]=(mu/d)nabla F_mu^B(theta), so the estimator is (1/d)nabla F_mu^B(theta), not nabla F_mu(theta). Because Thms B.4, B.5, and C.4 (Eq. (49)) all build on Thm B.3, the ZoAR variance and convergence theory is not established as stated.","section":"Appx C.4 / Thm B.3"},{"comment":"The importance-sampling identity in Eq. (10) is not valid for singular proposals. The Gaussian policy expectation is over R^d, so p must be a density with respect to Lebesgue measure. For u~Unif(S^{d-1}), the stated p(x)=1/Area(S^{d-1}(mu)) is a density with respect to surface measure, and for u~Unif({e_i}), p(x)=1/d is a density with respect to counting measure; neither is a Lebesgue density. Thus pi_theta(x)/p(x) in Eq. (11) is not a Radon-Nikodym derivative and the estimator does not estimate nabla J(theta) in the usual Monte Carlo sense. The gamma factors in Thm 3.3 are formal density ratios, and Corollary 3.4's learning-rate rescaling is unsupported for sphere and coordinate sampling. The statement should be restricted to absolutely continuous proposals or re-derived using a common reference measure.","section":"Appx C.3 / Thm 3.3"},{"comment":"The additive term B_2 in Eq. (49) is not shown to vanish under the stated choices eta=O(epsilon^2), 1-beta_2=O(epsilon^2). In the definition of B_2, the term sqrt(2/(beta_1(1-beta_2)))(1+beta_1)G contains G=2G_mu sqrt(zeta) sqrt(d[V+...]) with V=(sigma_xi^2+sigma_mu^2)/(NK mu^2) independent of eta, so this part of B_2 is of order sqrt(V)/epsilon as epsilon goes to 0 for fixed V. The remark after Thm B.6 says the bias can be small with a small learning rate eta, but the V-dependent part of B_2 is not controlled by eta. Consequently Eq. (49) does not establish convergence to a first-order stationary point, and the claim that query reuse enhances convergence is not supported by this theorem.","section":"Appx C.7 / Thm C.4"},{"comment":"In the proof of Thm B.5, step (a) replaces E||sum_{n,k} z_{n,k}||^2 with sum_{n,k} E||z_{n,k}||^2, where z_{n,k}=(f(theta_{t-n}+mu u_{n,k};xi)-b_t)u_{n,k}/mu. This equality requires the terms to be uncorrelated with zero mean; independence alone is not enough because E[z_{n,k}] is proportional to the smoothed gradient and is nonzero. The omitted cross terms are nonnegative, so the displayed expression is not an upper bound as written. The variance term V in Thm B.5 and the constants Xi and G in Thm C.4 are therefore not derived by the given proof.","section":"Appx C.6 / Eq. (29)"}],"minor_comments":[{"comment":"The second equality writes f(theta;xi) inside the expectation over x; it should be f(x;xi).","section":"Sec. 3.1, Eq. (6)"},{"comment":"The perturbation distribution used in the experiments is not specified; please state whether u~N(0,I_d) or another distribution is used, since Appx B assumes u~Unif(S^{d-1}).","section":"Sec. 5 and Appx D"},{"comment":"Because the compared ZOO and REINFORCE updates are algebraically identical by Thm 3.2 and Eq. (12), the numerical agreement in Fig. 2 is by construction; it should be presented as a sanity check of the implementation rather than as independent empirical validation of the equivalence.","section":"Appx E.1, Fig. 2"},{"comment":"Minor typos include \"gradeint\", \"proptotional\", \"schudule\", and \"retrive\"; these should be corrected.","section":"Appx D.1 and Appx F"},{"comment":"Reporting only five-run averages without error bands makes it hard to assess the significance of the claimed speedups; adding standard deviations or confidence intervals would help.","section":"Figs. 1 and 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main conceptual equivalence is largely definitional, so the publication value hinges on ZoAR's empirical demonstration and its theory. The Appx B flaws are substantial: the sphere-smoothing unbiasedness claim is wrong, the importance-sampling extension uses singular proposals improperly, and the formal convergence theorem has a nonvanishing B_2 floor. I would not reject outright because the algebraic identity and the empirical results are salvageable, but the current manuscript's theory section should not be published as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline equivalence is real but largely a re-labeling: with b(ξ)=f(θ;ξ), the REINFORCE estimator in (9) becomes the standard ZOO estimator in (2) by direct substitution, and Thm 3.1 is just reparameterization. That connection was already floating around in the ES/REINFORCE literature, so calling it “previously unrecognized” oversells it. What is genuinely new is ZoAR, and it deserves a serious look: the averaged baseline and query reuse are simple, sensible ideas, and the experiments show consistent gains across synthetic functions, adversarial attacks, and LLM fine-tuning.\n\nThe problem is the theory for ZoAR. The proof of Thm B.3 (Appx C.4) claims that for u~Unif(S^{d−1}) the estimator is unbiased for the smoothed gradient, citing Flaxman et al.'s Lemma 2.1. That lemma applies to ball smoothing and carries a factor 1/d. For the sphere-smoothed F_mu defined in Eq (3), the estimator is off by that factor: with F(θ)=a·θ you get E[(F(θ+µu)−F(θ))/µ · u] = a/d, not a. The proof literally redefines F_mu as ball smoothing, which contradicts Eq (3). That error propagates into Thm B.5 and the convergence bound. The B_2 term in Thm C.4 is also not shown to vanish: with the stated η=O(ε^2) and 1−β_2=O(ε^2), the constant V makes B_2 look like O(1/ε), so the bound as written is vacuous. And the experiments never state which u distribution was used, so it is unclear which theorem, if any, applies to the reported results.\n\nNone of this kills the practical contribution. ZoAR is a reasonable algorithm and the empirical results are encouraging. But the theoretical support as written is not sound. The paper should go to peer review, with the expectation of a major revision: fix or restrict the theory (e.g., to Gaussian perturbations, where Stein's lemma works), state the sampling distribution, provide error bars and code, and temper the novelty claims. If those are addressed, the conceptual framing could be a useful reference for connecting ZOO and policy gradient tools.","headline":"Gaussian ZOO=REINFORCE is a correct but mostly re-labeling insight; ZoAR shows promise, but its theoretical support has a concrete sphere-sampling error and a convergence bound that doesn't close.","tokens_in":36788,"tokens_out":8579,"would_cite":false,"duration_ms":82349,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C56","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that Gaussian-smoothed zeroth-order optimization is exactly single-step REINFORCE with a baseline, and uses that equivalence to build a faster ZOO algorithm, ZoAR, that averages recent function evaluations and reuses past…","keywords":["zeroth-order optimization","REINFORCE","policy gradient","baseline subtraction","query reuse","Gaussian smoothing","variance reduction","finite differences"],"falsifier":"Run ZoAR with history depth $N>1$ on a simple quadratic objective in $d=10$ with a fixed learning rate and compare the empirical mean-squared error of the gradient estimate to the bound of Theorem B.5; if the squared-bias term does not scale as predicted with $\\eta^2 d^2 (N-1)/(1-\\beta_2)$ and instead dominates the variance at practical settings, the bias-control claim fails. Separately, test the optimal-baseline formula $b_t^*=\\frac{1}{N}\\sum_{n=1}^N F_\\mu(\\theta_{t-n})$ under $u\\sim\\mathcal{N}(0,I_d)$: since the proof requires $\\|u\\|^2=1$, a direct variance computation with Gaussian perturbations would show whether the averaged baseline remains variance-minimizing or needs a distribution-dependent correction.","tokens_in":35551,"feed_emoji":"🎲","tokens_out":4279,"duration_ms":40913,"temperature":0.7,"pith_summary":"The paper proves that Gaussian-smoothed zeroth-order optimization (ZOO) is exactly single-step REINFORCE with a baseline, not merely analogous to it. On the objective side, the smoothed function that ZOO implicitly minimizes is identical to the expected reward of a single-step policy whose action is a Gaussian perturbation of the parameters. On the gradient side, the standard finite-difference estimator equals the REINFORCE estimator with the baseline set to the function value at the current parameters. If this equivalence is correct, the entire toolbox of policy-gradient variance reduction transfers directly to ZOO, and the paper exhibits one such transfer in the ZoAR algorithm, which uses an averaged baseline and query reuse to cut gradient variance and speed convergence.","feed_headline":"Gaussian ZOO is exactly single-step REINFORCE with a baseline","feed_subtitle":"The proof lets zeroth-order optimization borrow policy-gradient variance reduction, yielding the faster ZoAR algorithm.","key_machinery":"The load-bearing identity is the score-function form of the Gaussian perturbation: with $x=\\theta+\\mu u$ and policy $\\pi_\\theta(x)=\\mathcal{N}(\\theta,\\mu^2 I_d)$, one has $\\nabla_\\theta\\ln\\pi_\\theta(x)=(x-\\theta)/\\mu^2=u/\\mu$. Substituting $x_k=\\theta+\\mu u_k$ into the REINFORCE estimator $(9)$ turns it term-by-term into the ZOO finite-difference estimator $(2)$ when the baseline is $f(\\theta;\\xi)$, which is the paper's Theorem 3.2. The same substitution proves the objective equivalence (Theorem 3.1), and the importance-sampling extension (Theorem 3.3) computes the ratio $\\gamma=\\pi_\\theta(x_k)/p(x_k)$ for Gaussian, uniform-on-sphere, and coordinate-basis proposals, giving the learning-rate rescaling rule of Corollary 3.4.","core_discovery":"The central claim is that the Gaussian-smoothed ZOO objective $F_\\mu(\\theta)=\\mathbb{E}_{u}[F(\\theta+\\mu u)]$ equals the single-step policy optimization objective $J(\\theta)=\\mathbb{E}_{x\\sim\\pi_\\theta}[F(x)]$ when the policy is the reparameterized Gaussian $x=\\theta+\\mu u$ with $u\\sim\\mathcal{N}(0,I_d)$. Building on that, the paper proves the gradient estimator $\\hat\\nabla F(\\theta)=\\frac{1}{K}\\sum_k \\frac{f(\\theta+\\mu u_k;\\xi)-f(\\theta;\\xi)}{\\mu}u_k$ is identical to the REINFORCE estimator with Gaussian policy and baseline $b(\\xi)=f(\\theta;\\xi)$, revealing that the subtraction of $f(\\theta;\\xi)$ in ZOO is exactly a REINFORCE variance-reducing baseline. For non-Gaussian samplers, the paper extends the equivalence through importance sampling, obtaining a scaled identity $\\hat\\nabla_{\\mathrm{IS}}J(\\theta)=\\gamma\\hat\\nabla F(\\theta)$ with explicit $\\gamma$ factors, and shows that ZOO and REINFORCE achieve identical convergence when learning rates are scaled by $\\gamma$. These equivalences are the paper's first contribution; the second is ZoAR, which replaces the single-point baseline with an average of recent function values in a history buffer and reuses all buffered queries in the gradient estimate, with theorems bounding the resulting variance reduction, bias, and convergence.","pith_inferences":["A natural next step the authors leave implicit is to replace the stored-average baseline with a parameterized or learned estimate of the expected function value, effectively importing actor-critic ideas into ZOO; this could reduce bias when the objective landscape shifts quickly.","The theorem's optimal-baseline result is proven for $u\\sim\\mathrm{Unif}(S^{d-1})$ where $\\|u\\|^2=1$; for $u\\sim\\mathcal{N}(0,I_d)$ the same formula may not be exactly variance-minimizing, so the optimal baseline likely depends on the perturbation distribution, suggesting a distribution-specific baseline design.","The bias from query reuse grows with history depth $N$ and learning rate $\\eta$, so a linear or adaptive decay schedule for the history length, which the paper mentions as future work, would likely trade a small loss of variance reduction for a large gain in bias control.","If the equivalence holds for single-step REINFORCE, it should extend to multi-step settings: a ZOO estimator that reuses queries across several parameter updates is formally the same as a truncated or bootstrapped policy gradient, which could justify importing temporal-difference style smoothing into ZOO."],"forward_implications":["If ZOO estimators are REINFORCE estimators, then every variance-reduction technique developed for policy gradients, such as learned baselines, critics, or GAE-style advantage estimation, becomes a candidate for improving ZOO sample efficiency.","The baseline view explains why the subtraction of $f(\\theta;\\xi)$ helps in practice: it is not a finite-difference artifact but a provably variance-reducing REINFORCE baseline, and the averaged baseline in ZoAR is a Monte Carlo estimate of the variance-minimizing baseline $\\mathbb{E}[R(x)]$.","Query reuse, analogous to experience replay, increases the effective batch size without additional function queries, with a quantifiable bias-variance trade-off: larger history depth $N$ reduces variance but adds bias that scales with $\\eta^2 d^2 (N-1)/(1-\\beta_2)$, so a small learning rate keeps the bias small.","For non-Gaussian perturbation distributions, the importance-sampling scaling factors $\\gamma$ prescribe principled learning-rate rescaling, meaning a single ZOO implementation can switch sampling distributions while preserving the same effective convergence trajectory.","The convergence guarantee of ZoAR depends directly on the gradient variance $V=(\\sigma_\\xi^2+\\sigma_\\mu^2)/(NK\\mu^2)$, so any further reduction of $V$ yields a directly proportional speedup in the variance-dominated regime."],"supporting_citations":[{"why":"Supplies the Gaussian randomized finite-difference estimator and the smoothed objective $F_\\mu$ that the paper reinterprets as a single-step PO objective.","marker":"Nesterov & Spokoiny, 2017"},{"why":"Defines the REINFORCE estimator and its baseline variant, which the paper proves is identical to the ZOO gradient estimator.","marker":"Williams, 1992"},{"why":"Provides the policy gradient theorem that justifies the REINFORCE form for the Gaussian policy used in the equivalence.","marker":"Sutton et al., 1999"},{"why":"Supplies the standard result that baseline subtraction reduces gradient estimator variance, which the paper uses to reinterpret the ZOO subtraction.","marker":"Sutton & Barto, 2018"},{"why":"Uses a similar evolution-strategies estimator whose variance-reduction benefit the paper explains as REINFORCE-with-baseline, supporting the practical relevance of the equivalence.","marker":"Salimans et al., 2017"},{"why":"Introduces the uniform-on-sphere perturbation distribution whose importance-sampling scaling factor the paper computes in Theorem 3.3.","marker":"Flaxman et al., 2005a"},{"why":"Provides the ZO-AdaMM update rule that the paper pairs with its gradient estimator in experiments.","marker":"Chen et al., 2019"},{"why":"Defines the R-AdaZO update rule and the theoretical setting (assumptions, variance notation) that ZoAR's bias and variance theorems build on.","marker":"Shu et al., 2025b"},{"why":"Introduces ReLIZO, the query-reuse baseline that ZoAR is compared against empirically and whose approach of reusing previous queries is conceptually extended.","marker":"Wang et al., 2024"}],"fun_headline_variants":["Zeroth-order opt is secretly single-step policy optimization","ZOO equals REINFORCE with a baseline, new proof shows","Finite-difference ZOO is REINFORCE with a baseline","Policy-gradient baseline reduces ZOO variance","Zeroth-order optimization is one policy step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The algorithmic gain of ZoAR rests on the premise that the bias introduced by reusing historical queries stays small at practical learning rates; the paper's formal bound (Theorem C.4) leaves a constant floor $B_2$ depending on gradient variance, and the remark asserts without a quantitative guarantee that 'this bias can be small with a small learning rate $\\eta$.'","fun_headline_variants_meta":{"raw":{"variants":["Zeroth-order opt is secretly single-step policy optimization","ZOO equals REINFORCE with a baseline, new proof shows","Finite-difference ZOO is REINFORCE with a baseline","Policy-gradient baseline reduces ZOO variance","Zeroth-order optimization is one policy step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3655,"prompt_tokens":1106,"completion_tokens":2549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":2468}},"tokens_in":722,"tokens_out":2549,"duration_ms":23023,"temperature":1.0,"reasoning_tokens":2468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:52:58.571728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ZoAR with history depth $N>1$ on a simple quadratic objective in $d=10$ with a fixed learning rate and compare the empirical mean-squared error of the gradient estimate to the bound of Theorem B.5; if the squared-bias term does not scale as predicted with $\\eta^2 d^2 (N-1)/(1-\\beta_2)$ and instead dominates the variance at practical settings, the bias-control claim fails. Separately, test the optimal-baseline formula $b_t^*=\\frac{1}{N}\\sum_{n=1}^N F_\\mu(\\theta_{t-n})$ under $u\\sim\\mathcal{N}(0,I_d)$: since the proof requires $\\|u\\|^2=1$, a direct variance computation with Gaussian perturbations would show whether the averaged baseline remains variance-minimizing or needs a distribution-dependent correction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the REINFORCE estimator and its baseline variant, which the paper proves is identical to the ZOO gradient estimator."},{"cited_title":"S., McAllester, D","cited_arxiv_id":null,"evidence_quote":"Provides the policy gradient theorem that justifies the REINFORCE form for the Gaussian policy used in the equivalence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standard result that baseline subtraction reduces gradient estimator variance, which the paper uses to reinterpret the ZOO subtraction."},{"cited_title":"Relizo: Sample reusable linear interpolation-based zeroth-order optimization","cited_arxiv_id":null,"evidence_quote":"Introduces ReLIZO, the query-reuse baseline that ZoAR is compared against empirically and whose approach of reusing previous queries is conceptually extended."}],"review_version":1}