Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding synthetic trajectories to offline policy evaluation need not sacrifice statistically valid uncertainty estimates: two new methods, CP-Gen and DR-PPI, construct confidence intervals that cover the true policy value.

desk verdict CP-Gen's conformal band covers a new return difference, not the mean, so the state-conditioned interval does not follow; DR-PPI is a plausible PPI variant but the paper's combined claim is unsupported. read the letter →

arxiv 2507.20068 v3 pith:EHT2B7MH submitted 2025-07-26 cs.LG stat.ML

Aishwarya Mandyam , Jason Meng , Ge Gao , Jiankai Sun , Mac Schwager , Barbara E. Engelhardt , Emma Brunskill This is my paper · ORCID
classification cs.LGstat.ML
keywords off-policyevaluationconfidenceintervalsconformalpredictionprediction-poweredinferencedoublyrobustestimationdataaugmentationreinforcementlearningsynthetictrajectories
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes on a practical problem: when a reinforcement learning system is evaluated from logged data, can you add synthetic trajectories generated by a learned model without losing the ability to say how uncertain your value estimate is? The authors propose two estimators. CP-Gen builds a weighted conformal prediction band around the difference between real and synthetic returns for a fixed starting state, then adds that band to the simulator's estimate to obtain a confidence interval for the state-conditioned policy value $V^{\pi_e}(s)$. DR-PPI combines doubly robust estimation with prediction-powered inference to give an asymptotically valid confidence interval for the average policy value $V^{\pi_e}$. Across inventory, sepsis, HalfCheetah, and MIMIC-IV, both methods cover the ground-truth value at the requested level, whereas baselines that naively augment with synthetic data do not.

What carries the argument

The central mechanism is the paired return difference $\Delta_{rr'} = J(\tau)-J(\tilde{\tau})$, the difference between a real trajectory's return and a synthetic trajectory's return from the same initial state. CP-Gen assigns each such pair an epsilon-approximated importance weight $w_\epsilon(s,\delta_{rr'})$ that corrects the shift from behavior policy $\pi_b$ to target policy $\pi_e$ by averaging the product of inverse propensity score ratios over neighborhoods of the initial state and score; weighted conformal prediction on these scores produces the band in Equation (11). DR-PPI instead constructs a prediction-powered estimator whose correction term subtracts the mean of synthetic returns from the importance-weighted real return, then cross-fits the generative model over two data splits and forms a $z$-interval from a plug-in variance estimate. The former carries the state-conditioned guarantee; the latter carries the asymptotic population-level guarantee.

What would settle it

Consider an MDP with one state where the target return is $0$ with probability $0.975$ and $1000$ with probability $0.025$, while the synthetic return is identically $0$. The return difference is then $0$ with probability $0.975$ and $1000$ with probability $0.025$, so the conformal band at $\alpha=0.05$ collapses to $\{0\}$ in large samples, yet the true mean difference is $25$. Run CP-Gen on this MDP: if the band covers $97.5\%$ of new return differences but adding the band to the known synthetic mean produces an interval that misses the true $V^{\pi_e}(s)$, then the step from Equation (6) to Equation (7) is invalid.

Watch

Extended reading notes

Core claim

The paper's central claim is that uncertainty quantification survives the use of biased synthetic data, provided the bias is corrected rather than ignored. For CP-Gen, the key object is the return difference $\Delta_{rr'}$ between a behavior trajectory and a generated trajectory sharing the same initial state. Theorem 1 states that the weighted conformal band around this difference covers a new difference drawn under the target policy with probability at least $1-\alpha-\Delta_w$, where $\Delta_w$ is a coverage loss that shrinks as the epsilon-approximation radii and the number of samples improve. The paper then asserts that adding this band to the simulator estimate yields a valid confidence interval for $V^{\pi_e}(s)$. For DR-PPI, Theorem 2 states that the cross-fitted doubly robust estimator with a PPI-style correction is asymptotically normal, so the $z$-interval constructed from the plug-in variance has coverage at least $1-\alpha$ in the limit, even when the generative model is misspecified.

Load-bearing premise

The argument requires that a band which contains a randomly drawn return difference with high probability also contains the mean of that difference; a prediction interval for a single draw is not generally a confidence interval for the draw's expectation.

Editorial extensions

If this is right

  • A practitioner who has a biased generative model can still report a confidence interval for a policy's value, as long as the bias is corrected with one of these two constructions.
  • CP-Gen supplies intervals for state-conditioned value $V^{\pi_e}(s)$, which matters for settings like healthcare where the same treatment policy is applied to patients starting in the same state.
  • DR-PPI covers the population-average value $V^{\pi_e}$ and remains valid asymptotically even if the generative model is misspecified, provided the importance-sampling correction is consistent.
  • Naive augmentation, adding synthetic trajectories to IS or DR estimators without correction, fails in the paper's experiments, so the gain from synthetic data is not automatic; the correction is what buys validity.
  • In stochastic environments with accurate generative models, DR-PPI gives tighter intervals than standard IS; in deterministic or poorly modeled environments it matches IS, giving a practical rule to pick the narrower valid interval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical corollary not developed in the paper: the same $\epsilon$-approximation weights could be reused for conformal OPE without any synthetic data, by treating the generative model as optional rather than load-bearing.
  • The variance decomposition for DR-PPI points to a concrete selection rule: estimate the model-based variance and the correction-term variance from data and choose IS or DR-PPI by whichever is smaller, rather than always preferring the narrower observed interval.
  • Because CP-Gen conditions on the initial state, its construction could be applied to subpopulation-level evaluation, where the conditioning set is a clinically or administratively defined subgroup of initial states rather than a single state.
  • A natural testable extension is to plug a diffusion-based trajectory generator into both methods and measure whether the coverage loss $\Delta_w$ tracks the error in the generator's estimated transition dynamics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces two methods for constructing confidence intervals in off-policy evaluation (OPE) when synthetic trajectories generated by a learned model are available alongside real behavior data. CP-Gen is a weighted conformal prediction method intended to produce intervals for the state-conditioned policy value V^{π_e}(s) by forming a band over return differences between paired real and synthetic trajectories and adding it to a simulator estimate. DR-PPI is a cross-fitted doubly robust estimator inspired by prediction-powered inference, with a normal confidence interval for the average policy value V^{π_e}. The authors prove finite-sample coverage up to an approximation error for CP-Gen (Theorem 1) and asymptotic coverage for DR-PPI (Theorem 2), and report experiments on inventory control, sepsis, D4RL HalfCheetah, and MIMIC-IV.

Significance. If the guarantees were valid, the paper would provide the first principled uncertainty quantification for OPE with both real and synthetic data, a practically important setting. The DR-PPI construction is a reasonable, non-circular adaptation of prediction-powered inference, and the paper includes a broad empirical study across four domains, a real-world dataset, and a code release. The proofs are collected in an appendix, and the empirical comparisons cover several relevant baselines. However, the central CP-Gen claim rests on a mean-vs-draw conflation: a conformal band covering a new return difference does not yield a confidence interval for the expected return difference, and the paper's own experiments show CP-Gen below nominal coverage in three of four domains. Since this is the main theoretical contribution for state-conditioned values, the central claim of the paper is not established.

major comments (3)
  1. [Section 3.1, Eqs. (6)-(7); Theorem 1, Eq. (21)] Theorem 1 establishes coverage for a new return difference Δ_{rr'} under P^{π_e}. The interval in Eq. (7) is formed by adding the conformal band to a simulator mean, so valid coverage of V^{π_e}(s) would require P(E[Δ|s] ∈ Ĉ_{n,α}(s)) ≥ 1-α. Coverage of a draw does not imply coverage of its conditional mean. For example, if Δ|s puts 0.9 mass at -1 and 0.1 mass at 101, a 90% conformal band equal to {-1} covers new draws with probability 0.9 but excludes the mean 9.2; the corresponding interval for V^{π_e}(s) would not cover. This gap is not addressed by the Δ_w term, which only controls weight-estimation error, and it is consistent with Table 4, where CP-Gen achieves 92-94% coverage instead of 95% in three domains.
  2. [Section 3.1, Remark following Eq. (7)] The remark claims that because V^π(s) is itself an expectation, marginal coverage is sufficient. This is incorrect: marginal coverage of a new draw is a statement about the predictive distribution, not about the location of the expectation. The target of inference, E[Δ|s], is a functional of the conditional distribution, and no conformal band built from individual scores targets that functional under the stated assumptions. The proof of Theorem 1 therefore does not support the paper's stated interval for V^{π_e}(s).
  3. [Section 4.2, Eq. (16), and Appendix E.5] The proof of Theorem 2 and the variance derivation condition on the fitted generative models f1 and f2 and do not quantify the randomness of the generative-model fit. Consistency of the plug-in variance estimator in Eq. (16) is asserted, not proved, and the derivation in Appendix E.5 produces a conditional variance rather than establishing the unconditional CLT used for the cross-fitted estimator. As a result, the asymptotic coverage guarantee in Eq. (24) is not established at the stated level of generality.
minor comments (5)
  1. [Section 4.2 heading] The heading contains a typo: 'confidnece' should be 'confidence'.
  2. [Section 3.1, text after Eq. (5)] The phrase 'where n/M is the number of behavior/synthetic trajectories' should read 'where n and M are the numbers of behavior and synthetic trajectories, respectively'.
  3. [Section 5.3] The symbol Cips is used in the discussion of Sepsis coverage but is only defined in Appendix E.2; it should be defined at first use in the main text.
  4. [Table 4] Coverage rates are reported without standard errors or confidence intervals for the coverage estimates; given the small numbers of iterations (25 or 50), this makes it difficult to assess whether deviations from 95% are statistically meaningful.
  5. [Appendix B] The code is said to be provided on GitHub, but no repository URL or access information is given in the manuscript.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: CP-Gen calibrates on held-out return differences and DR-PPI is a cross-fitted PPI estimator; the main concern is a missing implication (draw coverage vs. mean coverage), not circular fitting.

full rationale

CP-Gen's conformal band is computed from return differences Δrr' using weights estimated on a held-out calibration split; it is not constructed from V^{πe}(s), so Theorem 1 is not an input that has been renamed as a prediction. DR-PPI is built from a cross-fitted PPI estimator with an importance-sampling correction, and its asymptotic normality is taken from Angelopoulos et al. (2023) using standard IS/WIS identities, not from the paper's own conclusions. The paper's self-citations (Mandyam et al. 2024; Gao et al. 2024; Gao et al. 2023) motivate the use of synthetic trajectories and describe the VAE architecture, but none of them supplies the coverage argument. The paper itself notes that clipping introduces unaccounted bias (Section 3.3), and Table 4 shows below-nominal CP-Gen coverage in three domains; the remark that marginal coverage of a draw suffices for a mean is an unsupported logical step between Theorem 1 and Eq. (7), and is a correctness/rigor issue rather than a circular derivation. Because no fitted parameter is renamed as a prediction and no theorem reduces by construction to its own assumptions, the circularity score is low.

Assumptions & free parameters 3 free parameters · 10 assumptions · 0 invented entities

DR-PPI rests on standard boundedness, support, and fixed-model assumptions plus a known initial state distribution. CP-Gen adds two ad hoc regularity conditions, Lipschitz score smoothness and bounded density, to make the epsilon approximation tractable. The most serious problem is not an axiom entry but a missing implication: the paper never proves that a prediction band for a random return difference can serve as a confidence interval for the mean policy value.

free parameters (3)
  • epsilon_s (CP-Gen) = not reported
    Radius for the state neighborhood in the epsilon-approximation of conformal weights. It is chosen via cross-validation per environment and directly controls the coverage loss Delta_w in Theorem 1, making it a fitted knob with a direct effect on the claimed validity.
  • epsilon_r (CP-Gen) = not reported
    Radius for the return-difference neighborhood in the weight approximation. Like epsilon_s, it is a hand-tuned hyperparameter that enters the coverage bound and is not reported for the experiments.
  • IPS clipping constant (DR-PPI) = n^{1/2}
    The importance ratios are clipped at n^{1/2} following Ionides (2008). This is a user-set constant that introduces finite-sample bias in the confidence interval, and the paper explicitly states the bias is not accounted for in the theory.
assumptions (10)
  • domain assumption Assumption 4: common support, pi_e(a|s) > 0 implies pi_b(a|s) > 0
    Standard OPE assumption used in Appendix E.2 to ensure importance ratios are well defined.
  • domain assumption Assumption 5: bounded returns 0 <= J(tau) <= C_r
    Standard boundedness condition used in the proofs of both theorems.
  • domain assumption Assumption 6: bounded importance ratios c_ips <= pi_e/pi_b <= C_ips
    Standard bounded-importance-weight condition used throughout the proofs.
  • domain assumption Assumptions 1-2: Lipschitz policies and Lipschitz transition dynamics
    Introduced in Section 4.1 to control the error of the epsilon approximation in CP-Gen.
  • ad hoc to paper Assumption 3: score smoothness, w is L_r-Lipschitz in the return difference
    Introduced specifically for the epsilon approximation; if the importance weight is not Lipschitz in the score, the stated coverage-loss bound does not hold.
  • ad hoc to paper Assumption 7: bounded density p_min <= p(s, delta) <= p_max for (S, Delta) under pi_b
    A regularity condition added in Appendix E.2 to control local sample sizes in the covering argument; not verified in the empirical domains.
  • domain assumption Initial state distribution d0 is known for DR-PPI
    Stated in Section 3.2; in experiments the initial state distribution is taken from empirical cohorts rather than a known analytic distribution.
  • domain assumption Behavior policy pi_b is known in the theory
    Stated in Section 2.2; in the experiments pi_b is estimated, and robustness is tested only in one inventory setting.
  • domain assumption The generative model distribution is treated as fixed
    The theory conditions on the learned model and does not account for the uncertainty from fitting the generative model; the model-bias term is treated as a fixed discrepancy.
  • standard math Weighted conformal prediction background results
    CP-Gen relies on the weighted conformal framework of Tibshirani et al. and on Proposition 2 of Foffano et al.; the proof of Theorem 1 invokes these as established results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data." pith.science (2026). https://pith.science/paper/EHT2B7MH

@misc{pith2026250720068,
  author       = {Pith},
  title        = {Pith review of: PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EHT2B7MH}},
  note         = {Machine review of arXiv:2507.20068}
}
abstract

Off-policy evaluation (OPE) methods estimate the value of a new reinforcement learning (RL) policy prior to deployment. Recent advances have shown that leveraging auxiliary datasets, such as those synthesized by generative models, can improve the accuracy of OPE methods. Unfortunately, such auxiliary datasets may also be biased, and existing methods for using data augmentation within OPE lack principled uncertainty quantification. In high stakes domains like healthcare, reliable uncertainty estimates are important for ensuring safe and informed deployment of RL policies. In this work, we propose two methods to construct valid confidence intervals for OPE with data augmentation. The first provides a confidence interval over $V^{\pi}(s)$, the policy value conditioned on an initial state $s$. To do so we introduce a new conformal prediction method suitable for Markov Decision Processes (MDPs) with continuous state spaces, extending prior work to higher-dimensional settings. Second, we consider the more common task of estimating the average policy performance over all initial states, $V^{\pi}$; we introduce a method that draws on ideas from doubly robust estimation and prediction powered inference. Across simulators spanning inventory management, robotics, healthcare, and a real healthcare dataset from MIMIC-IV, we find that our methods can effectively leverage auxiliary data and consistently produce confidence intervals that cover the ground truth policy values, unlike previously proposed methods. Our work enables a future in which OPE can provide rigorous uncertainty estimates for high-stakes domains.

Figures

Figures reproduced from arXiv: 2507.20068 by the authors.

Figure 1
Figure 1. DR-PPI and CP-Gen are robust to annotation quality and improve in quality as the number of generated trajectories increase in the Inventory and Sepsis environments. (Left) We fix N (i.e., number of behavior trajectories) for both the Inventory (N = 60) and Sepsis (N = 200) settings. We alter the number of generated trajectories from 20 ∗ N to 500 ∗ N. We report the coverage rate across 50 iterations for α = 0.05. (R… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. X4Val: Learning Neural Surrogates for Variance-Reduced Policy Evaluation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    X4Val learns transferable neural predictors from non-paired multi-domain data and incorporates them into control-variates estimators to reduce variance in real-world robotic policy evaluation by up to 38.4%.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1880]

    For methods that produce an interval, we report the interval forα= 0.05and whether the interval covers the true policy value

    ✗ (1964.35, 2004.42) ✓ MIMIC-IV 1 0.689 (0,1) ✓ (-1.28, 0.92) ✓ (0.977, 1.1012) ✓ Table 6:CP-Genoutperforms baselines across domains with continuous state-spaces, producing conformal prediction intervals that cover the true policy value,Vπe(s). For methods that produce an interval, we report the interval forα= 0.05and whether the interval covers the true ...

    work page 2023

  2. [2014]

    CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

    International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450327381. Aishwarya Mandyam, Shengpu Tang, Jiayu Yao, Jenna Wiens, and Barbara E. Engelhardt. Candor: Coun- terfactual annotated doubly robust off-policy evaluation.ArXiv, abs/2412.08052, 2024. Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online markov deci...

    work page Pith review arXiv 2024

  3. [2016]

    Assumption 4(Common support).π e(a|s)>0→π b(a|s)>0,∀s∈S,∀a∈A

    to support our theoretical results. Assumption 4(Common support).π e(a|s)>0→π b(a|s)>0,∀s∈S,∀a∈A. Assumption 5(Bounded return).0≤J(τ)≤C r for allτ∼p. Assumption 6(Bounded IPS weights).c ips≤ πe(a|s) πb(a|s)≤C ips,∀s∈S,∀a∈A. These assumptions are standard in the literature and minimally restrictive, thus enabling the analysis of CP-Gen’s performance under ...

    work page 2011

  4. [2510]

    ✓ (-1566.32, -1045.68) ✗ (-4449.27, 1082.33) ✓ Sepsis -0.40 -0.12 (-1,0) ✓ (-0.01, 0.01) ✗ (-1.36, 0.54) ✓ D4RL Half Cheetah 1990.39 1393.98 (1750,

    work page 1990

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.