{"id":"cb9d7763-2d2d-4049-8af7-5068c3e07f8a","arxiv_id":"2507.20068","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"CP-Gen and DR-PPI are two estimators that aim to provide valid confidence intervals for off-policy evaluation with synthetic auxiliary data, with asymptotic guarantees for DR-PPI and empirical coverage for both.","lead":"This paper proposes two methods to construct confidence intervals for off-policy reinforcement learning evaluation when real logs are supplemented by synthetic trajectories. It reports empirical coverage across inventory, sepsis, robot, and ICU datasets, which matters for safer deployment in high-stakes settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 bounds coverage of a new return difference, not of the policy-value mean; Eqs. (6)-(7) therefore do not establish a confidence interval for V^{π_e}(s).","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: CP-Gen's stated conformal guarantee is for a new return difference, while the paper's central claim is a confidence interval for the mean policy value. I agree with that diagnosis. The paper's step from Eq. (6) to Eq. (7) is a non sequitur. A conformal prediction set for a draw does not generally contain the draw's expectation; the paper's Remark in Section 3.1 ('marginal coverage is sufficient') states the opposite of what is needed. This is not a technicality resolvable by smaller Δ_w: even with exact weights and zero approximation error, the mean of Δ can lie outside the conformal band. The empirical results do not repair the gap, and the paper's own Table 4 shows CP-Gen below the requested 95% coverage in Sepsis, D4RL, and MIMIC-IV. DR-PPI may be salvageable, but the paper's combined claim — that both methods produce valid CIs — rests on the CP-Gen inference-target argument, so the verdict should remain REJECT.","tokens_in":28889,"tokens_out":5949,"duration_ms":66736,"concrete_test":"Construct a small MDP with an initial state s and known transition/reward distributions such that Δ = J(τ)-J(τ̃) has mean outside its central (1-α) interval under π_e (e.g., a two-point or mixture return difference with a rare large positive value). Implement CP-Gen using the true simulator and exact weights (ϵ_s=ϵ_r=0, so Δ_w≈0), and over many replications compute (i) empirical coverage of a new Δ and (ii) coverage of the true V^{π_e}(s) by the interval from Eq. (7). If (i) is ≥1-α while (ii) is below 1-α, the inference-target mismatch is confirmed; if (ii) is also ≥1-α, the concern is refuted.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim requires that the CP-Gen interval in Eq. (7) cover V^{π_e}(s). Theorem 1 establishes P^{π_e}(Δ_{rr'} ∈ Ĉ_{n,α}(S)) ≥ 1-α-Δ_w for a new draw of the return difference, and Eq. (7) forms μ̃_sim + Ĉ_{n,α}(s). A valid confidence interval for V^{π_e}(s) = μ̃_sim + E[Δ_{rr'}|s] requires coverage of the conditional mean of Δ, i.e., P(E[Δ|s] ∈ Ĉ_{n,α}(s)) ≥ 1-α, which is not implied by coverage of a new draw. The text even inverts this: the Remark states that 'the target is V^{π}(s), which is itself an expectation, so marginal coverage is sufficient' — marginal coverage of a draw does not imply containment of an expectation. Concretely, if Δ|s has a mixture distribution with 90% mass at -1 and 10% at 101, a 90% conformal band can be {-1}, which covers new draws 90% of the time but excludes the mean 9.2, so the corresponding interval for V(s) would not cover. This gap is not repaired by the ϵ-approximation or by Δ_w; those quantities control weight estimation error, not the mean-vs-draw mismatch. Table 4's below-nominal CP-Gen coverage in three of four domains is consistent with the gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two methods for constructing confidence intervals in off-policy evaluation (OPE) when synthetic trajectories generated by a learned model are available alongside real behavior data. CP-Gen is a weighted conformal prediction method intended to produce intervals for the state-conditioned policy value V^{π_e}(s) by forming a band over return differences between paired real and synthetic trajectories and adding it to a simulator estimate. DR-PPI is a cross-fitted doubly robust estimator inspired by prediction-powered inference, with a normal confidence interval for the average policy value V^{π_e}. The authors prove finite-sample coverage up to an approximation error for CP-Gen (Theorem 1) and asymptotic coverage for DR-PPI (Theorem 2), and report experiments on inventory control, sepsis, D4RL HalfCheetah, and MIMIC-IV.","tokens_in":29205,"tokens_out":8057,"duration_ms":76773,"significance":"If the guarantees were valid, the paper would provide the first principled uncertainty quantification for OPE with both real and synthetic data, a practically important setting. The DR-PPI construction is a reasonable, non-circular adaptation of prediction-powered inference, and the paper includes a broad empirical study across four domains, a real-world dataset, and a code release. The proofs are collected in an appendix, and the empirical comparisons cover several relevant baselines. However, the central CP-Gen claim rests on a mean-vs-draw conflation: a conformal band covering a new return difference does not yield a confidence interval for the expected return difference, and the paper's own experiments show CP-Gen below nominal coverage in three of four domains. Since this is the main theoretical contribution for state-conditioned values, the central claim of the paper is not established.","major_comments":[{"comment":"Theorem 1 establishes coverage for a new return difference Δ_{rr'} under P^{π_e}. The interval in Eq. (7) is formed by adding the conformal band to a simulator mean, so valid coverage of V^{π_e}(s) would require P(E[Δ|s] ∈ Ĉ_{n,α}(s)) ≥ 1-α. Coverage of a draw does not imply coverage of its conditional mean. For example, if Δ|s puts 0.9 mass at -1 and 0.1 mass at 101, a 90% conformal band equal to {-1} covers new draws with probability 0.9 but excludes the mean 9.2; the corresponding interval for V^{π_e}(s) would not cover. This gap is not addressed by the Δ_w term, which only controls weight-estimation error, and it is consistent with Table 4, where CP-Gen achieves 92-94% coverage instead of 95% in three domains.","section":"Section 3.1, Eqs. (6)-(7); Theorem 1, Eq. (21)"},{"comment":"The remark claims that because V^π(s) is itself an expectation, marginal coverage is sufficient. This is incorrect: marginal coverage of a new draw is a statement about the predictive distribution, not about the location of the expectation. The target of inference, E[Δ|s], is a functional of the conditional distribution, and no conformal band built from individual scores targets that functional under the stated assumptions. The proof of Theorem 1 therefore does not support the paper's stated interval for V^{π_e}(s).","section":"Section 3.1, Remark following Eq. (7)"},{"comment":"The proof of Theorem 2 and the variance derivation condition on the fitted generative models f1 and f2 and do not quantify the randomness of the generative-model fit. Consistency of the plug-in variance estimator in Eq. (16) is asserted, not proved, and the derivation in Appendix E.5 produces a conditional variance rather than establishing the unconditional CLT used for the cross-fitted estimator. As a result, the asymptotic coverage guarantee in Eq. (24) is not established at the stated level of generality.","section":"Section 4.2, Eq. (16), and Appendix E.5"}],"minor_comments":[{"comment":"The heading contains a typo: 'confidnece' should be 'confidence'.","section":"Section 4.2 heading"},{"comment":"The phrase 'where n/M is the number of behavior/synthetic trajectories' should read 'where n and M are the numbers of behavior and synthetic trajectories, respectively'.","section":"Section 3.1, text after Eq. (5)"},{"comment":"The symbol Cips is used in the discussion of Sepsis coverage but is only defined in Appendix E.2; it should be defined at first use in the main text.","section":"Section 5.3"},{"comment":"Coverage rates are reported without standard errors or confidence intervals for the coverage estimates; given the small numbers of iterations (25 or 50), this makes it difficult to assess whether deviations from 95% are statistically meaningful.","section":"Table 4"},{"comment":"The code is said to be provided on GitHub, but no repository URL or access information is given in the manuscript.","section":"Appendix B"}],"recommendation":"reject","confidential_remarks":"The CP-Gen defect is not a local technical issue; it is the method's main theoretical claim and would require a new inferential construction rather than a repair of the present derivation. The manuscript is marked as published in TMLR, which is unusual for an arXiv submission under review, but this does not affect the technical assessment above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The problem is real: off-policy evaluation with synthetic auxiliary data needs confidence intervals, and no one has done this for MDPs. The paper's main method, CP-Gen, has a proof gap that matters: Theorem 1 shows the conformal band covers a new return difference Δ with probability 1-α-Δ_w, but the claimed interval is for V^{π_e}(s), a mean of that difference. A band that covers a new draw does not generally contain the draw's expectation. The paper's remark says marginal coverage is sufficient because the target is an expectation; that inverts the logic. A mixture distribution where 90% of draws are -1 and 10% are 101 has a 90% conformal band {-1} that covers draws but misses the mean 9.2. The Δ_w bound only accounts for weight estimation error, so it cannot close this gap. The empirical coverage below nominal in three of four domains is consistent with this.\n\nWhat is genuinely good: DR-PPI is a sane adaptation of prediction-powered inference with an importance-sampling correction, and its asymptotic argument is concise and likely right. The ε-approximation is a credible mechanism for weighted conformal OPE in continuous state spaces. The experiments are broad, including a real MIMIC-IV cohort, and baselines are handled carefully. The paper is transparent about limitations and ships code.\n\nSoft spots in proportion. DR-PPI's plug-in variance estimator is not shown to be consistent, and the randomness of the learned generative model is conditioned away; the cross-fitting independence is asserted rather than proved. These are real but moderate issues, likely fixable with standard assumptions. The IPS clipping constant is noted as an unaccounted bias, which is minor if the rate is n^{1/2}. None of this is as damaging as the CP-Gen gap, because the gap is load-bearing for the first claim.\n\nRecommendation: this paper deserves a serious referee. The topic is important, DR-PPI may be a useful contribution, and the empirical work is solid. But the central claim, as stated, is unsupported. I would send it to review and ask the authors to either derive a genuine confidence interval for V^{π_e}(s) or to restrict CP-Gen's guarantee to something the theorem actually supports. That is a substantial revision, not a desk reject.","headline":"CP-Gen's conformal band covers a new return difference, not the mean, so the state-conditioned interval does not follow; DR-PPI is a plausible PPI variant but the paper's combined claim is unsupported.","tokens_in":29731,"tokens_out":4250,"would_cite":false,"duration_ms":42856,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding synthetic trajectories to offline policy evaluation need not sacrifice statistically valid uncertainty estimates: two new methods, CP-Gen and DR-PPI, construct confidence intervals that cover the true policy value.","keywords":["off-policy evaluation","confidence intervals","conformal prediction","prediction-powered inference","doubly robust estimation","data augmentation","reinforcement learning","synthetic trajectories"],"falsifier":"Consider an MDP with one state where the target return is $0$ with probability $0.975$ and $1000$ with probability $0.025$, while the synthetic return is identically $0$. The return difference is then $0$ with probability $0.975$ and $1000$ with probability $0.025$, so the conformal band at $\\alpha=0.05$ collapses to $\\{0\\}$ in large samples, yet the true mean difference is $25$. Run CP-Gen on this MDP: if the band covers $97.5\\%$ of new return differences but adding the band to the known synthetic mean produces an interval that misses the true $V^{\\pi_e}(s)$, then the step from Equation (6) to Equation (7) is invalid.","tokens_in":28655,"feed_emoji":"🎯","tokens_out":15233,"duration_ms":135367,"temperature":0.7,"pith_summary":"This paper takes on a practical problem: when a reinforcement learning system is evaluated from logged data, can you add synthetic trajectories generated by a learned model without losing the ability to say how uncertain your value estimate is? The authors propose two estimators. CP-Gen builds a weighted conformal prediction band around the difference between real and synthetic returns for a fixed starting state, then adds that band to the simulator's estimate to obtain a confidence interval for the state-conditioned policy value $V^{\\pi_e}(s)$. DR-PPI combines doubly robust estimation with prediction-powered inference to give an asymptotically valid confidence interval for the average policy value $V^{\\pi_e}$. Across inventory, sepsis, HalfCheetah, and MIMIC-IV, both methods cover the ground-truth value at the requested level, whereas baselines that naively augment with synthetic data do not.","feed_headline":"Synthetic data need not ruin confidence intervals for policy value","feed_subtitle":"Two estimators mix real and generated trajectories while keeping coverage guarantees, enabling safer RL evaluation.","key_machinery":"The central mechanism is the paired return difference $\\Delta_{rr'} = J(\\tau)-J(\\tilde{\\tau})$, the difference between a real trajectory's return and a synthetic trajectory's return from the same initial state. CP-Gen assigns each such pair an epsilon-approximated importance weight $w_\\epsilon(s,\\delta_{rr'})$ that corrects the shift from behavior policy $\\pi_b$ to target policy $\\pi_e$ by averaging the product of inverse propensity score ratios over neighborhoods of the initial state and score; weighted conformal prediction on these scores produces the band in Equation (11). DR-PPI instead constructs a prediction-powered estimator whose correction term subtracts the mean of synthetic returns from the importance-weighted real return, then cross-fits the generative model over two data splits and forms a $z$-interval from a plug-in variance estimate. The former carries the state-conditioned guarantee; the latter carries the asymptotic population-level guarantee.","core_discovery":"The paper's central claim is that uncertainty quantification survives the use of biased synthetic data, provided the bias is corrected rather than ignored. For CP-Gen, the key object is the return difference $\\Delta_{rr'}$ between a behavior trajectory and a generated trajectory sharing the same initial state. Theorem 1 states that the weighted conformal band around this difference covers a new difference drawn under the target policy with probability at least $1-\\alpha-\\Delta_w$, where $\\Delta_w$ is a coverage loss that shrinks as the epsilon-approximation radii and the number of samples improve. The paper then asserts that adding this band to the simulator estimate yields a valid confidence interval for $V^{\\pi_e}(s)$. For DR-PPI, Theorem 2 states that the cross-fitted doubly robust estimator with a PPI-style correction is asymptotically normal, so the $z$-interval constructed from the plug-in variance has coverage at least $1-\\alpha$ in the limit, even when the generative model is misspecified.","pith_inferences":["A practical corollary not developed in the paper: the same $\\epsilon$-approximation weights could be reused for conformal OPE without any synthetic data, by treating the generative model as optional rather than load-bearing.","The variance decomposition for DR-PPI points to a concrete selection rule: estimate the model-based variance and the correction-term variance from data and choose IS or DR-PPI by whichever is smaller, rather than always preferring the narrower observed interval.","Because CP-Gen conditions on the initial state, its construction could be applied to subpopulation-level evaluation, where the conditioning set is a clinically or administratively defined subgroup of initial states rather than a single state.","A natural testable extension is to plug a diffusion-based trajectory generator into both methods and measure whether the coverage loss $\\Delta_w$ tracks the error in the generator's estimated transition dynamics."],"forward_implications":["A practitioner who has a biased generative model can still report a confidence interval for a policy's value, as long as the bias is corrected with one of these two constructions.","CP-Gen supplies intervals for state-conditioned value $V^{\\pi_e}(s)$, which matters for settings like healthcare where the same treatment policy is applied to patients starting in the same state.","DR-PPI covers the population-average value $V^{\\pi_e}$ and remains valid asymptotically even if the generative model is misspecified, provided the importance-sampling correction is consistent.","Naive augmentation, adding synthetic trajectories to IS or DR estimators without correction, fails in the paper's experiments, so the gain from synthetic data is not automatic; the correction is what buys validity.","In stochastic environments with accurate generative models, DR-PPI gives tighter intervals than standard IS; in deterministic or poorly modeled environments it matches IS, giving a practical rule to pick the narrower valid interval."],"supporting_citations":[{"why":"Supplies the conformal OPE construction for MDPs that CP-Gen extends, and the proof of Theorem 1 follows Proposition 2 from this work.","marker":"Foffano et al. (2023)"},{"why":"Supplies weighted conformal prediction under covariate shift, the machinery behind the sample reweighting in CP-Gen.","marker":"Tibshirani et al. (2019)"},{"why":"First applies weighted conformal prediction to OPE in contextual bandits; CP-Gen extends its scoring and weighting idea to MDPs with synthetic data.","marker":"Taufiq et al. (2022)"},{"why":"Supplies prediction-powered inference, whose estimator form DR-PPI adapts for the correction term and whose Proposition 1 underlies Theorem 2.","marker":"Angelopoulos et al. (2023)"},{"why":"Supplies the cross-fitting argument used to split the behavior dataset so the generative model and the correction term are independent.","marker":"Chernozhukov et al. (2018)"},{"why":"Defines importance sampling for off-policy evaluation, used to reweight real trajectories in DR-PPI.","marker":"Precup et al. (2000)"},{"why":"Provides standard OPE assumptions, such as common support and bounded importance ratios, that the theoretical results rely on.","marker":"Thomas & Brunskill (2016)"},{"why":"Provides the sepsis simulator used to test both methods' empirical coverage.","marker":"Oberst & Sontag (2019)"},{"why":"Provides the D4RL HalfCheetah environment used for the continuous-control evaluation.","marker":"Fu et al. (2020)"},{"why":"Supplies the MIMIC-IV EHR data used for the real-world empirical evaluation.","marker":"Johnson et al. (2024)"}],"fun_headline_variants":["Confidence intervals survive biased synthetic data","Bias-corrected OPE: valid intervals with synthetic data","Two estimators keep coverage despite biased aux data","Synthetic data bias fixed: trustworthy policy value intervals","RL evaluation with rigorous intervals using augmented data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument requires that a band which contains a randomly drawn return difference with high probability also contains the mean of that difference; a prediction interval for a single draw is not generally a confidence interval for the draw's expectation.","fun_headline_variants_meta":{"raw":{"variants":["Confidence intervals survive biased synthetic data","Bias-corrected OPE: valid intervals with synthetic data","Two estimators keep coverage despite biased aux data","Synthetic data bias fixed: trustworthy policy value intervals","RL evaluation with rigorous intervals using augmented data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2822,"prompt_tokens":1000,"completion_tokens":1822,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":1750}},"tokens_in":616,"tokens_out":1822,"duration_ms":13005,"temperature":1.0,"reasoning_tokens":1750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:51:40.116056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Consider an MDP with one state where the target return is $0$ with probability $0.975$ and $1000$ with probability $0.025$, while the synthetic return is identically $0$. The return difference is then $0$ with probability $0.975$ and $1000$ with probability $0.025$, so the conformal band at $\\alpha=0.05$ collapses to $\\{0\\}$ in large samples, yet the true mean difference is $25$. Run CP-Gen on this MDP: if the band covers $97.5\\%$ of new return differences but adding the band to the known synthetic mean produces an interval that misses the true $V^{\\pi_e}(s)$, then the step from Equation (6) to Equation (7) is invalid.","supporting_citations":[],"review_version":1}