{"id":"caa5c285-1093-4783-a01e-a327981c7b18","arxiv_id":"2505.13809","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper derives the efficient influence function for optimal policy values in MDPs, proves it does not exist under policy non-uniqueness, and proposes a sequential doubly robust estimator that provides valid conservative inference in such regimes.","lead":"A statistics paper proposes new confidence intervals for the value of the best possible policy in a Markov decision process, using offline data collected under a different policy. It handles cases where the best policy is not unique, a situation that breaks standard methods, and stays efficient when the best policy is unique.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption A.6 as stated (κ_Q+κ_ω≥1/2) is not enough for Theorem 5.1: the proof needs a strict product rate >1/2, so the lower-bound CLT can fail at the boundary.","rationale":"The central claim is that NSAVE provides efficiency under uniqueness and valid conservative lower bounds under possible non-uniqueness, with the inference theory running through Theorem 5.1 and Corollary 5.2. Display (29) is the only place where the nuisance rates enter the remainder, and the proof's transition from the product bound to o_P0(j^{-(1/2+ε)}) is exactly where the boundary rate fails. This is not a question of external consensus: it is an internal mismatch between the stated assumption (≥1/2) and the proof's requirement (>1/2). At equality, the bias term contributes at the same order as the martingale noise, so the studentized statistic is asymptotically biased and the lower bound can overstate the optimal value. Because every subsequent theorem (5.3, 5.4, 5.6) inherits Theorem 5.1 or uses the same step, this single gap is load-bearing for the main inference claims. The fix is straightforward—state κ_Q+κ_ω>1/2 or prove an extra cancellation—so the appropriate verdict remains conditional. I agree with the reader's identification; the reader explicitly noted the need for a strict product rate greater than j^{-1/2}. I did not elevate other concerns (variance estimator construction, missing code, dataset mismatch) because they are secondary relative to this proof gap.","tokens_in":60002,"tokens_out":5854,"duration_ms":54682,"concrete_test":"Check the boundary case analytically: set κ_Q=κ_ω=1/4 and re-derive the step in Appendix D.1. Using (29), the cumulative bias term is (N-ℓ_N)^{-1} Σ_{j=ℓ_N+1}^N c_j j^{-1/2}; with c_j bounded away from zero, this is Θ((N-ℓ_N)^{-1/2}), not o((N-ℓ_N)^{-1/2}). Then σ^{-1}_{R1N} R_CLB,1N has asymptotic mean 2c, so coverage is below 1-α. A small simulation with nuisance estimators deliberately calibrated to exact n^{-1/4} rates and a fixed/known policy sequence would confirm the shift; if the mean shift vanishes, the proof needs to exhibit the missing cancellation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing defect is in Assumption A.6 together with display (29). Theorem 5.1 states the result under κ_Q+κ_ω≥1/2, but Appendix D.1 requires the cumulative bias after (29) to be o_P0(j^{-(1/2+ε)}) so that the remainder in (30) is negligible. The product bound (29) yields a bias of order j^{-(κ_Q+κ_ω)}. At the allowed boundary κ_Q+κ_ω=1/2 this is O_P0(j^{-1/2}), not o_P0(j^{-1/2}); consequently (N-ℓ_N)^{-1} Σ_{j=ℓ_N+1}^N j^{-1/2} = Θ((N-ℓ_N)^{-1/2}), exactly the CLT scale. The normalized R_CLB,1N in (31) then has a first-order bias, so the Gaussian limit in Theorem 5.1 and the nominal coverage of the conservative lower bound in Corollary 5.2 are not guaranteed. The assumption needs the strict inequality κ_Q+κ_ω>1/2, or an additional condition forcing the product bias to decay faster than j^{-1/2}. Separately, no primitive conditions are supplied to verify even the stated rates for the sequentially re-estimated nuisances under policy feedback, but the boundary inconsistency is already visible within the proof.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies semiparametric inference for the value of an optimal policy in Markov decision processes, allowing for the possibility that the optimal policy is non-unique. It first characterizes when the efficient influence function for the optimal value functional exists: under a unique deterministic optimal policy it coincides with the fixed-policy EIF evaluated at the optimal policy (Theorem 3.1), while under unrestricted optimal rules no influence function exists (Theorem 3.2). The central methodological contribution is NSAVE, a sequentially re-weighted estimator built from trajectory-level doubly robust estimating functions. The paper proves a conservative lower confidence bound for the optimal value using only non-positivity of regret and a martingale CLT, with no regularity conditions on the estimated policy sequence (Theorem 5.1, Corollary 5.2), and a two-sided CLT under a margin-type condition (Theorems 5.3–5.5). It also provides a smoothed one-step estimator (Theorem 6.1) and a post-selection inference template (Corollary 6.2), together with simulations and an OhioT1DM application.","tokens_in":60392,"tokens_out":4294,"duration_ms":44363,"significance":"If the main results hold, the paper makes a useful contribution to off-policy inference for optimal policies. The conservative lower-bound construction in Theorem 5.1 is the most distinctive idea: by exploiting the non-positive regret term, the authors obtain valid one-sided inference while avoiding any conditions on the convergence of the estimated policy sequence. This is a genuine improvement over existing SAVE-type inference, which requires non-degeneracy and margin conditions that break down for deterministic or nearly deterministic optimal policies. The semiparametric efficiency claim for the unique-optimum case (Corollary 5.5, Theorem 5.6) and the smoothing extension also address a real gap in the literature. The finite-state analysis and the explicit double-robustness decomposition are presented in considerable detail. At the same time, the paper's central claims rest on Assumption A.6, which is not currently sufficient for the theorem as stated, and on sequential nuisance rate assumptions that are not verified by any primitive conditions. These issues are load-bearing but appear fixable within the paper's scope.","major_comments":[{"comment":"Assumption A.6 states only κ_Q + κ_ω ≥ 1/2, but the proof of Theorem 5.1 in Appendix D.1 needs the cumulative residual bias to be o_P0(j^{-(1/2+ε)}), as written after display (29). The product bound (29) gives a bias of order j^{-(κ_Q+κ_ω)}. At the allowed boundary κ_Q + κ_ω = 1/2, this is O_P0(j^{-1/2}), not o_P0(j^{-1/2}), so the weighted average of biases is of order (N-ℓ_N)^{-1/2}, exactly the CLT scale. Consequently the normalized statistic σ^{-1}_{R1N} R_{CLB,1N} in (31) can carry a first-order bias, and the Gaussian limit in Theorem 5.1 and the coverage guarantee in Corollary 5.2 are not ensured. The assumption should be strengthened to the strict inequality κ_Q + κ_ω > 1/2, or an additional condition must force the product bias to decay faster than j^{-1/2}.","section":"Section 5.1, Assumption A.6 and display (29)"},{"comment":"No primitive conditions are supplied that guarantee the j^{κ_Q}- and j^{κ_ω}-consistency rates for the sequentially re-estimated nuisances when the target policy itself is re-estimated from the same growing sample at every step. This feedback between policy updates and nuisance estimation can invalidate standard cross-fitting rate results, and Assumption A.6 simply postulates the rates that the proof needs. Since every central limit theorem in Section 5 rests on this assumption, the paper should either provide verifiable sufficient conditions for the required sequential rates or identify existing results that cover this feedback structure.","section":"Assumption A.6 (Section 5)"},{"comment":"Assumption A.13 controls the Gateaux derivatives D_QL and D_ωL at the empirical distribution P_{τ(j-1)T}, but the proof of Theorem 5.6 uses Assumption A.13 to conclude that the population-level total gap Gap_total defined below display (41) is O_P0(j^{-κ_L}). These objects are different, and the proof does not provide the empirical-process uniformity argument needed to convert one into the other. This gap is load-bearing for the double-robustness claim, because the bound on ||ω(·;π̂^ω) - ω(·;π̂^Q)|| depends on the population total gap. The authors should either restate Assumption A.13 in terms of the population distribution P0, or add a lemma showing that the empirical and population gaps are asymptotically equivalent under the already-imposed Donsker and boundedness conditions.","section":"Theorem 5.6 and Assumption A.13"}],"minor_comments":[{"comment":"The statement uses the symbol ω_Q in the rate condition ('ω_Q > 1/2 + max{...}') but the assumption is otherwise about κ_Q; please clarify the notation and state the condition in terms of κ_Q.","section":"Theorem 6.1"},{"comment":"The simulation study reports empirical coverage based on only 100 Monte Carlo replications, so a nominal 95% coverage estimate has a Monte Carlo standard error of roughly 2.2 percentage points; the coverage comparisons for SAVE and NSAVE should be interpreted with this noise level in mind, and the paper should say so.","section":"Section 7"},{"comment":"Table 1 reports log MSE values without standard errors or confidence intervals; reporting Monte Carlo uncertainty for the MSE estimates would strengthen the comparison, especially for the very large negative log-MSE values attributed to NSAVE in Scenario C.","section":"Table 1"},{"comment":"The ω-based estimated optimal policy π̂^(ω) is introduced and used in notation, but the text immediately says it is not needed for implementation; a brief note explaining that it only serves as a bookkeeping device for the ratio estimator would avoid confusion.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for math.ST and the core idea is promising, but the boundary issue in Assumption A.6 and the missing primitive conditions on sequential nuisance rates are substantive rather than cosmetic. The proof of Theorem 5.6 also appears to mix empirical and population versions of the saddle-point gap. All three issues are fixable, but they require real work, so I recommend major revision rather than rejection. I did not find evidence of circularity or inappropriate benchmarking against the existing literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper is serious and fills a recognized gap: valid off-policy inference for the optimal policy value in MDPs when the argmax is non-unique. The conservative lower-bound construction is the genuine contribution—using the non-positive regret term to get one-sided validity without any regularity on the estimated policy sequence is clever and, once you fix a rate condition, sound. The EIF existence/non-existence characterization (Theorems 3.1–3.2) is new, and the NSAVE estimator's efficiency claim under uniqueness is plausible. The martingale proof is detailed and the double-robustness argument is real.\n\nThe soft spots are real but mostly repairable. The stress-test concern lands: Assumption A.6 states κ_Q+κ_ω ≥ 1/2, but the proof of Theorem 5.1 needs the product bias to be o_P(j^{-(1/2+ε)}). At the boundary, the residual bias is O_P(j^{-1/2}) and the normalized sum in (31) picks up a first-order bias term, so the Gaussian limit and Corollary 5.2's coverage are not guaranteed. The fix is simple—require strict inequality, or add a condition forcing the product bias to decay faster—but as stated the theorem is false. Relatedly, the paper gives no primitive conditions for the sequentially re-estimated nuisances to achieve the assumed rates under policy feedback; that is a bigger gap, though common in this literature. The online variance estimator is assumed consistent (A.8) with no construction; the windowed sample variance mentioned in Section 4.2 is not enough for the theory. There is also no code, and the abstract says Drink Less while the body uses OhioT1DM; that inconsistency should have been caught. These are all revision-level issues.\n\nThe simulation evidence is supportive but not overwhelming: 100 replications, and the implementation of the variance estimator in the experiments is not described. Still, the paper's central idea is strong and the flaws are fixable. I would send it to peer review, expecting major revision on the rate conditions and clarity. I'd bring it to a reading group and would cite it if my work touches optimal policy inference.","headline":"A serious methodological paper with a genuinely useful conservative lower-bound construction, but the proof of Theorem 5.1 assumes a strict rate product while the stated assumption only gives the boundary, so the CLT fails there.","tokens_in":60874,"tokens_out":3541,"would_cite":true,"duration_ms":33986,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G05","62G20","62M05","90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that the value of an optimal policy in a Markov decision process can be estimated with valid confidence intervals even when the optimal policy is not unique, using a sequential estimator whose one-sided inference…","keywords":["off-policy evaluation","optimal policy value","semiparametric efficiency","non-regular inference","Markov decision processes","doubly robust estimation","post-selection inference","martingale central limit theorem"],"falsifier":"Run the paper's structurally misspecified scenario with a deliberately constructed plateau of exactly tied optimal actions and measure the empirical product of the estimated Q and importance-ratio errors at each sequential step; if that product fails to shrink faster than $j^{{-1/2}}$ while the policy is re-estimated, Theorem 5.1's central limit theorem should fail, with the conservative lower bound's coverage dropping below its nominal level and the two-sided interval failing to shrink at root-n rate.","tokens_in":59748,"feed_emoji":"🎯","tokens_out":8431,"duration_ms":74784,"temperature":0.7,"pith_summary":"This paper asks how to draw confidence intervals for the value of the best policy when the best policy itself is learned from data and may not be unique. It establishes that in the regular case of a unique, deterministic optimal policy, the optimal policy value has an efficient influence function equal to the fixed-policy influence function evaluated at that policy, and its NSAVE estimator attains the semiparametric efficiency bound. In the irregular case, where several optimal policies tie or the argmax is nearly deterministic, the paper shows no influence function exists, yet NSAVE still delivers asymptotically valid conservative lower bounds and, under mild margin conditions, two-sided intervals. This bears directly on offline reinforcement learning and mobile-health decision support, where the best policy is learned from historical data and ties among good actions are common.","feed_headline":"NSAVE gives valid bounds even when the optimal policy is non-unique","feed_subtitle":"A martingale-based estimator keeps valid coverage when competing policies tie, where standard influence-function methods break down.","key_machinery":"The carrying object is the trajectory-level efficient influence function (EIF), the minimal-variance influence curve for a fixed policy evaluated on an entire trajectory, re-estimated sequentially and averaged with inverse-variance weights. NSAVE re-estimates the Q function and the marginal importance-sampling ratio from the growing sample, and the studentized trajectory scores form a martingale difference array, so a martingale central limit theorem applies. The key bias term is a product: the conditional bias of each trajectory score is bounded by the product of the estimation errors of Q and the importance ratio, and Assumption A.6 requires this product to shrink faster than the square root of the sample size, making root-n inference possible. The error decomposition separates the martingale statistical error from the non-positive policy-value regret, which is why the conservative lower bound needs no conditions on the estimated policy sequence.","core_discovery":"The central discovery is that the non-regularity of the optimal policy value is governed by whether competing optimal policies have distinct first-order gradients. When the optimal policy is unique and deterministic, the value functional is pathwise differentiable and its efficient influence function coincides with the fixed-policy influence function; NSAVE then inherits double robustness and semiparametric efficiency. When the optimal policy is non-unique, the functional has no influence function at all, so no regular asymptotically linear estimator exists; NSAVE's validity instead comes from a martingale decomposition that controls the sequential bias without requiring the estimated policy sequence to converge at a prescribed rate, and it provides conservative lower bounds as well as two-sided intervals under an added margin condition. The smoothing construction extends this by replacing the argmax with a softmax, and the post-selection construction reports uniform confidence sets over the plausible winners.","pith_inferences":["The sequential inverse-variance averaging device suggests a general template: any estimator built from trajectory-level estimating functions under data-adaptive policies will be root-n consistent when the nuisance error product shrinks faster than the square root of the sample size, a recipe that may transfer to other irregular functionals such as dynamic treatment regimes.","Because the lower-bound result does not require the estimated policy to converge, it could support safe offline deployment decisions: a deliberately conservative bound on the optimal value is enough to decide whether a learned policy is better than the behavior policy, and the bound's width can be monitored as data accumulate.","A testable extension would replace Assumption A.6's rate product with an empirical stopping rule that keeps adding trajectory evaluations until the estimated product of the Q and importance-ratio errors falls below the root-n threshold, converting a rate assumption into an adaptive procedure."],"forward_implications":["For a unique deterministic optimal policy, NSAVE attains the semiparametric efficiency bound and retains double robustness without requiring a linear Q-function model.","When the optimal policy is non-unique, NSAVE returns conservative lower confidence bounds with asymptotic coverage at least one minus the nominal level, even if the estimated policy sequence has no limiting distribution.","Under a margin-type condition and sufficient nuisance rates, two-sided root-n confidence intervals for the optimal value are available, and the smoothing estimator reaches the same efficiency bound under uniqueness with only two nuisance fits.","The post-selection procedure reports confidence sets that simultaneously cover the values of all empirically selected optimal policies and shrink to the oracle interval when the optimum is unique."],"supporting_citations":[{"why":"SAVE is the baseline sequential estimator whose linear-model and non-degeneracy restrictions motivate NSAVE, and whose sequential evaluation framework NSAVE revises.","marker":"Shi et al. (2022)"},{"why":"Supplies the fixed-policy efficient influence function and the minimax weight learning construction used to estimate the marginal importance ratio.","marker":"Uehara et al. (2020)"},{"why":"Provides the double-robustness expansion used to bound the sequential bias term as a product of nuisance estimation errors.","marker":"Kallus & Uehara (2022)"},{"why":"Source for non-unique optimal treatment regimes and the non-regularity phenomena that the paper transfers to Markov decision processes.","marker":"Luedtke & Van Der Laan (2016)"},{"why":"Source of the softmax smoothing and first-order de-biasing technique that the paper extends from single-period to dynamic settings.","marker":"Whitehouse et al. (2025)"},{"why":"Supplies the two-step multiple-winners inference framework used for the post-selection confidence sets.","marker":"Petrou-Zeniou & Shaikh (2024)"},{"why":"Provides the martingale limit theory underlying the central limit theorem for the weighted trajectory scores.","marker":"Hall & Heyde (2014)"},{"why":"Gives the pathwise differentiability, Riesz representation, and non-regularity results used to characterize existence and non-existence of the efficient influence function.","marker":"Van Der Vaart (2000)"}],"fun_headline_variants":["NSAVE provides valid inference even when optimal policies are non-unique","Martingale method yields valid bounds for non-unique optimal policies","New NSAVE handles tied optimal policies with double robustness","Semiparametric inference for optimal policy values without uniqueness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The inference collapses if the sequentially re-estimated nuisance functions Q and the importance ratio do not, together, converge faster than the square root of the sample size while the optimal policy is being re-estimated from the same growing data; the paper assumes this combined rate (Assumption A.6) but gives no primitive conditions that guarantee it survives the feedback between policy updates and nuisance estimation.","fun_headline_variants_meta":{"raw":{"variants":["NSAVE provides valid inference even when optimal policies are non-unique","Martingale method yields valid bounds for non-unique optimal policies","New NSAVE handles tied optimal policies with double robustness","Semiparametric inference for optimal policy values without uniqueness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1495,"prompt_tokens":995,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":611,"tokens_out":500,"duration_ms":4890,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:57.943229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's structurally misspecified scenario with a deliberately constructed plateau of exactly tied optimal actions and measure the empirical product of the estimated Q and importance-ratio errors at each sequential step; if that product fails to shrink faster than $j^{{-1/2}}$ while the policy is re-estimated, Theorem 5.1's central limit theorem should fail, with the conservative lower bound's coverage dropping below its nominal level and the two-sided interval failing to shrink at root-n rate.","supporting_citations":[],"review_version":1}