Concept-driven Off Policy Evaluation

Jack Teversham; Ritam Majumdar; Sonali Parbhoo

REVIEW 4 major objections 5 minor 27 references

Concept-driven Off Policy Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Concept-based importance sampling preserves unbiasedness and cuts variance in off-policy evaluation.

desk verdict Concept-based OPE is a promising framing, but the central unbiasedness proof assumes the wrong behavior policy and the variance induction has a sign error, so the headline guarantees do not hold as written. read the letter →

arxiv 2411.19395 v1 pith:KJOHXJJ7 submitted 2024-11-28 stat.ML cs.AIcs.LG

Ritam Majumdar , Jack Teversham , Sonali Parbhoo This is my paper · ORCID

classification stat.MLcs.AIcs.LG

keywords off-policyevaluationimportancesamplingconceptbottleneckmodelsinterpretableconceptsvariancereductionreinforcementlearninginterventions

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing the state in an importance-sampling ratio with an interpretable concept—a coarse grouping of states that a human can understand—and claims this makes off-policy evaluation more reliable and more explainable. For known concepts, it proves that the resulting CIS and CPDIS estimators stay unbiased under the completeness assumption, and that their variance is no larger than that of standard IS, PDIS, and marginalized-IS estimators when a per-timestep covariance condition holds. For unknown concepts, it learns concepts end-to-end through a concept bottleneck model whose loss includes the OPE variance itself; experiments in a gridworld and on ICU patient data show variance reductions of one to three orders of magnitude at the cost of some bias. The paper also shows that intervening on high-variance learned concepts, guided by domain knowledge, can further reduce bias or variance.

What carries the argument

The central object is the concept map $\phi: S \to C$ that sends each state to an interpretable concept vector $c_t$, together with the concept-conditioned policies $\pi^c_e$ and $\pi^c_b$ used in the importance ratio $\rho^c_{0:t}=\prod_{t'=0}^t \pi^c_e(a_{t'}|c_{t'})/\pi^c_b(a_{t'}|c_{t'})$. The variance comparison is carried by the covariance condition $\mathrm{Cov}(\rho^c_{0:t}r_t, \rho^c_{0:k}r_k) \le \mathrm{Cov}(\rho_{0:t}r_t, \rho_{0:k}r_k)$, and the proofs adapt a state-abstraction lemma showing the per-concept variance difference is nonpositive. For unknown concepts, the machinery is a concept bottleneck model whose bottleneck output is trained with a combined loss: next-state prediction, L1 interpretability, cosine diversity, a policy-closeness term enforcing the $\beta$ deviation bound, and the OPE variance itself.

What would settle it

Run a synthetic MDP with a known evaluation policy and a behavior policy that depends on a state feature not captured by any concept map $\phi$, then compute the CIS estimator with oracle concepts on an arbitrarily large batch; if the estimate does not converge to the true $V_{\pi_e}$, the unbiasedness theorem's data-generation premise is violated.

Watch

Extended reading notes

Core claim

The paper's central claim is that importance ratios conditioned on a concept bottleneck, $\pi^c_e(a_t|c_t)/\pi^c_b(a_t|c_t)$ instead of $\pi_e(a_t|s_t)/\pi_b(a_t|s_t)$, preserve the unbiasedness of importance sampling while lowering variance whenever concepts group states with similar action distributions. Theorem 5.3 states unbiasedness for both CIS and CPDIS under the completeness assumption; Theorem 5.4 states the variance comparison to IS and PDIS under a covariance inequality; Theorem 5.5 states the comparison to marginalized importance sampling; and Theorem 5.6 says the Cramer-Rao MSE bound tightens by $K^{2T}$, where $K$ is the ratio of concept-cardinality to state-cardinality. For learned concepts, Theorem 6.1 concedes finite bias because the change of measure from $\pi_b$ to $\pi^c_b$ is no longer available, while the variance bounds survive under the same covariance condition. The unbiasedness proof in Appendix D.1.1 takes its first expectation under the concept-conditioned behavior policy $\pi^c_b$, so the telescope result is conditional on the data actually being generated by a concept-conditioned policy.

Load-bearing premise

The load-bearing premise is that the batch data were collected by a policy that chooses actions using only the concept label, not the full state; only then does the concept-based importance weight telescope to give unbiasedness, and the proof in Appendix D.1.1 assumes exactly this in its first expectation.

Editorial extensions

If this is right

If the theorems hold, any OPE pipeline can replace state-conditioned importance ratios with concept-conditioned ones and, under the completeness and covariance conditions, inherit unbiasedness with lower variance.
The $K^{2T}$ tightening of the MSE bound makes concept coarseness a design lever: coarser concept spaces improve worst-case sample complexity exponentially in the horizon.
The learned-concept algorithm offers a variance-aware concept discovery procedure that, in the paper's experiments, beats human-defined concepts in both WindyGridworld and MIMIC-III while remaining interpretable.
Concept-based interventions give practitioners a concrete mechanism: replace or remove a high-variance concept and re-run the estimator, instead of treating OPE as a black box.
The imperfect-concept ablation shows concept quality is load-bearing; concepts that violate the desiderata can perform worse than state-based IS, so the desiderata function as necessary conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the unbiasedness claim is to check whether real logged data satisfy the concept-conditioned behavior-policy premise; if they do not, the observed bias in Figure 2 is the expected price, and practitioners should reweight or relabel data so that it holds.
The covariance condition in Theorems 5.4 and 5.5 can be estimated from logged trajectories; a validation study on hospital data would show how often concept groupings actually satisfy it, which is the empirical question the paper leaves open.
Because the learned-concept loss minimizes variance but not bias (the true on-policy value is unavailable), a natural extension is to add a model-based or doubly robust bias term to the loss, which could recover unbiasedness without sacrificing interpretability.
The $K^{2T}$ factor points to a trade-off: coarser concepts shrink variance but may violate the policy-closeness assumption, so concept learning should monitor the deviation bound alongside the OPE metric.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Short version: the core idea—doing OPE through interpretable concepts and learning those concepts with a CBM—is genuinely worth discussing, but the main theoretical guarantee is not proven for the setting the paper defines. Section 3 says the batch data come from a state-based behavior policy π_b(a|s), yet the proof of Theorem 5.3 in Appendix D.1.1 computes expectations under the concept-based policy π^c_b(a|c) and cancels ratios that only telescope if the data distribution factors through concepts. Assumption 5.1 does not bridge that gap. Unsurprisingly, Figure 2 shows bias for known concepts even with oracle concepts; that is not finite-sample noise, it is the theorem failing against the paper's own setup.

What is genuinely new: the end-to-end CBM concept learner with an OPE-variance loss (Algorithm 1) and the intervention framework. The known-concept CIS estimator, however, is mathematically the same as abstraction-based importance sampling (Pavse and Hanna, 2022b), which the paper cites but does not flag as a direct overlap. The experiments are reasonably extensive, the code is available, and the empirical variance reductions beyond bias look real. The imperfect-concepts ablation is a nice touch.

The soft spots beyond the bias proof are serious. The variance comparison in Appendix D.1.3 has a sign error in the induction: substituting C ≤ sqrt(SC′) gives a lower bound on D′, not an upper bound, and the proof concludes D′ ≤ 0 where D′ ≥ 0 was needed. Theorems 5.4 and 5.5 put the desired covariance inequality as an assumption, so they are conditional statements rather than derived guarantees. For unknown concepts, Algorithm 1 directly optimizes the variance it later reports; the bias is acknowledged but not quantified, and the Appendix E bounds are loose.

Bottom line: the framing and the empirical story are intriguing, but the load-bearing theory is broken. I would still send this to a knowledgeable referee rather than desk-reject—there is enough of a kernel that a major revision could produce something solid. As it stands, the central claims should not be cited.

If you work on OPE, engage with it; otherwise skim the experiments and the intervention section.

Referee Report

4 major / 5 minor

Summary. This paper proposes replacing the state in importance-sampling (IS) ratios for off-policy evaluation (OPE) with human-interpretable concepts, defining concept-based IS (CIS) and concept-based per-decision IS (CPDIS) estimators. For known concepts it claims unbiasedness (Theorem 5.3) and lower variance relative to ordinary IS, PDIS, and MIS under covariance conditions (Theorems 5.4 and 5.5), and it develops an end-to-end algorithm (Algorithm 1) that learns parameterized concepts via a concept bottleneck model with a loss that includes the OPE estimator's variance. Experiments on WindyGridworld and MIMIC-III report large variance and MSE reductions for concept-based estimators, while also reporting higher bias even for known oracle concepts, and an intervention analysis is used to interpret and adjust learned concepts.

Significance. If the theoretical claims were correct, the paper would make a useful contribution: grouping states into interpretable concepts could reduce OPE variance while providing interpretability and targeted interventions. The paper also has strengths: it provides a code link, detailed appendices, a real-world medical dataset, an ablation with imperfect concepts, and a comparison with state abstractions. However, the central unbiasedness theorem is proved under a different sampling model than the one defined in the paper, and the reported experiments directly contradict it. The variance-reduction proof contains a sign error, and the covariance conditions in the variance theorems largely restate the desired conclusion. As a result, the main theoretical and empirical claims are not currently established.

major comments (4)

[Section 3, Definitions 4.1-4.2, Appendix D.1.1, Theorem 5.3] The proof of Theorem 5.3 computes the bias under the concept-based behavior policy π^c_b: line (a) of Appendix D.1.1 writes E_{π^c_b}[V-hat^{CIS}], and line (c) writes the trajectory probability as ∏_t π^c_b(a_t|c_t), so the cancellation in step (e) is valid only when trajectories are generated by π^c_b. But Section 3 defines the batch data as generated by the state-based policy π_b(a|s), and Assumption 5.1 only asserts support/completeness; it does not imply π_b(a|s) = π^c_b(a|ϕ(s)) for all (s,a). Unbiasedness for V_{π_e} requires E_{τ∼π_b}[ρ^c_{0:T} ∑ γ^t r_t] = V_{π_e}, which the proof never establishes. This is load-bearing because Theorem 5.3 is the paper's headline guarantee. The contradiction is visible in Figure 2, where known-concept estimators, including oracle concepts, show nonzero bias; Section 5.3 attributes this to poor coverage, but that explanation is not consistent with the claimed proof under Assumption 5.1.
[Appendix D.1.3, Equations (h)-(m)] The induction proof that the concept-based IS ratio has lower variance than the state-based IS ratio has a sign error. The goal is to show D' ≥ 0, but after substituting C ≤ √(SC') into D' = (SC' − C)X²/(C'Y), the text derives D' ≤ 0 and then states that the induction hypothesis holds. Moreover, the claim that SC' − √(SC') is never negative is false when 0 ≤ SC' < 1. Thus the variance-ratio comparison is not proven, and this invalidates the ratio-level inequality used in the variance comparisons of Theorems 5.4 and 5.5.
[Theorems 5.4 and 5.5] The covariance assumptions in Theorems 5.4 and 5.5 are effectively restatements of part of the desired variance inequality. The theorem states that if Cov(ρ^c_{0:t}r_t, ρ^c_{0:k}r_k) ≤ Cov(ρ_{0:t}r_t, ρ_{0:k}r_k), then the concept-based estimator has lower variance; but this covariance condition is not derived from the concept mapping or from Assumptions 5.1-5.2. Without a structural argument showing that grouping states into concepts implies the condition, the theorem is a conditional identity rather than a substantive variance-reduction guarantee. When rewards are fixed to 1, the condition reduces exactly to the ratio-variance comparison in Appendix D.1.3, whose proof fails.
[Algorithm 1, Line 12; Section 6] The unknown-concept section is a central claimed contribution, but Algorithm 1 uses the OPE estimator's variance on the training trajectories as a training loss (Line 12), then reports results on a held-out split. This is not circular in the narrow sense of evaluating on the same data, but the paper provides no analysis of how this variance-minimizing training affects the bias-variance trade-off, and Theorem 6.1 concedes that unknown-concept estimators are biased. Given that the experiments show higher bias for learned concepts, the claim that the algorithm produces better OPE estimators needs a more careful treatment of the bias introduced by the concept-learning objective rather than only an empirical variance reduction.

minor comments (5)

[Assumption 5.2] Assumption 5.2 states the same inequality twice: both conditions read |π^c_e(a|c) − π_e(a|s)| < β. The second inequality should presumably involve π^c_b and π_b.
[Definitions 4.1 and 4.2] The concept-based behavior policy π^c_b(a|c) and evaluation policy π^c_e(a|c) are used in the estimators but are never formally defined in Section 4; the paper should state how they are derived from π_b, π_e, and the concept mapping ϕ.
[Theorem 5.6 and Appendix D.1.6] The notation K is used inconsistently: Theorem 5.6 describes K as the ratio of the cardinality of the concept space to the state space, whereas Appendix D.1.6 defines K = K1/K2 through counts of states mapping to a concept under π_e and π_b. A single consistent definition is needed.
[Section 5.3, Figure 2] The text says known-concept estimators are 'unbiased' in theory but reports higher bias in practice, even for oracle concepts. This discrepancy is not resolved in the paper and should be discussed explicitly, especially because it bears on the validity of Theorem 5.3.
[Appendix G.1] The three-stage training strategy is described in prose only; providing a precise schedule or pseudocode for when the OPE-metric loss is activated would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main guarantees are conditional derivations, and learned-concept results are evaluated on a held-out split.

full rationale

No step in the paper reduces by construction to its own input. Theorem 5.3 is a standard importance-sampling telescoping proof: the cancellation in Appendix D.1.1 is the usual identity E_{π^c_b}[ρ^c f] = E_{π^c_e}[f], which holds by definition of ρ^c. The paper does not define the data-generating π_b to equal π^c_b, nor does it define V_{π_e} as the concept-policy value, so any failure of unbiasedness for the state-based target is an assumption gap rather than a circular equation. The variance theorems (5.4, 5.5, 6.2, 6.3) are explicitly conditional on a covariance inequality; the conclusion also requires the per-timestep variance comparison, which the appendix attempts separately, so the assumption is not simply the conclusion restated. For learned concepts, Algorithm 1 trains on Ttrain and reports metrics on the held-out TOPE split (lines 2-15), so the reported variance reductions are not forced on the evaluation set by the line-12 OPE-metric loss. Citations to Pavse & Hanna (2022b) and Liu et al. (2020) are external; the only self-citations (Majumdar et al. 2023, Havasi et al. 2022, Wu et al. 2022) are background or future-work remarks and are not load-bearing. The paper honestly flags the trajectory-distribution mismatch for learned concepts and the higher bias observed in practice. Correctness concerns about the π^c_b sampling assumption in the unbiasedness proof and the sign error in the D.1.3 induction are real, but they are flaws in the proof, not circular reductions of the claimed result to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on five free parameters or hand-chosen mappings and on several assumptions, most importantly the covariance inequalities that are close to the desired conclusions. No new physical or model entities are introduced; concepts are deterministic functions of states, not additional latent structures.

free parameters (5)

CBM concept weights = Tables 2 and 3 (WindyGridworld/MIMIC)
Learned by gradient descent with the OPE variance in the loss; central to the unknown-concept results.
Loss weighting coefficients in Algorithm 1 = not reported
Relative weights of L_output, L_interpretability, L_diversity, L_OPE-metric, L_policy are not specified; the three-stage training adds thresholds that are also unspecified.
Concept dimension d = 4
Chosen by hand for both domains; affects concept capacity and variance.
Beta in Assumption 5.2 = 0 in experiments
Practitioner-defined tolerance between concept and state policies; set to 0 for all experiments.
Known concept mappings = WindyGridworld: 25 hand-defined values; MIMIC: 15 vital signs binned into 10 levels
Hand-designed concept functions are selected by the authors, not learned; the choice affects all known-concept results.

assumptions (5)

domain assumption Completeness: if pi_b(a|s) and pi^c_b(a|c) are positive then pi_e(a|s) and pi^c_e(a|c) are positive (Assumption 5.1).
Needed for the IS ratios to be well-defined and for the change of measure; not always satisfied in finite batch data.
ad hoc to paper Beta-closeness of concept and state policies (Assumption 5.2).
Imposes that concept policies are close to state policies, but the paper does not justify why this holds or how beta is set in practice.
ad hoc to paper Covariance condition in Theorem 5.4: Cov(ρ^c_{0:t}r_t, ρ^c_{0:k}r_k) <= Cov(ρ_{0:t}r_t, ρ_{0:k}r_k).
This is the central inequality needed for variance reduction; the paper provides no proof that concepts satisfy it, only asserts interpretability helps design policies to align with it.
ad hoc to paper Covariance condition in Theorem 5.5 comparing to MIS.
Similarly assumed, not derived.
standard math Change of measure lemma from Pavse and Hanna (2022b): E_{c∼dπ^c} f(c) = E_{s∼dπ} f(φ(s)).
Valid when concepts are deterministic functions of states and distributions are pushforwards; used extensively in the proofs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Concept-driven Off Policy Evaluation." pith.science (2026). https://pith.science/paper/KJOHXJJ7

@misc{pith2026241119395,
  author       = {Pith},
  title        = {Pith review of: Concept-driven Off Policy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJOHXJJ7}},
  note         = {Machine review of arXiv:2411.19395}
}

read the original abstract

Evaluating off-policy decisions using batch data poses significant challenges due to limited sample sizes leading to high variance. To improve Off-Policy Evaluation (OPE), we must identify and address the sources of this variance. Recent research on Concept Bottleneck Models (CBMs) shows that using human-explainable concepts can improve predictions and provide better understanding. We propose incorporating concepts into OPE to reduce variance. Our work introduces a family of concept-based OPE estimators, proving that they remain unbiased and reduce variance when concepts are known and predefined. Since real-world applications often lack predefined concepts, we further develop an end-to-end algorithm to learn interpretable, concise, and diverse parameterized concepts optimized for variance reduction. Our experiments with synthetic and real-world datasets show that both known and learned concept-based estimators significantly improve OPE performance. Crucially, we show that, unlike other OPE methods, concept-based estimators are easily interpretable and allow for targeted interventions on specific concepts, further enhancing the quality of these estimators.

Figures

Figures reproduced from arXiv: 2411.19395 by the authors.

**Figure 2.** WindyGridworld: Known Concept-based estimators have lower variance, MSE, higher [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Inverse propensity score comparisons under concepts and states. We observe the frequency [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: For both domains, unknown concept-based estimators show lower variance. In WindyGrid [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Interpretation of Optimized Concepts. WindyGridworld: The first two subplots compare [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Interventions: Qualitative interventions reduce Bias and MSE for unknown estimators in [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Schematic of windy-gridworld environment. The top-right corner refers to the goal target [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]

**Figure 8.** Figure 8: Comparison of learned concepts with state abstraction clusters: The first two subplots show [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]

**Figure 9.** Figure 9: Imperfect concepts baseline. We consider a scenario where the concepts are just function of [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]

**Figure 10.** Figure 10: Inverse propensity score comparisons under concepts and states. Column 1 represents IPS [PITH_FULL_IMAGE:figures/full_fig_p037_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages

[1]

(b) The second term is always greater than 0 (c) Applying Bernstein inequality with probability 1- δ

) + vuut ln( 2 δ ) N 3 − N 2 NX i<j (X 2 i − X 2 j )2 (d) Explanation of steps: (a) We begin with the definition of variance. (b) The second term is always greater than 0 (c) Applying Bernstein inequality with probability 1- δ. Xi refers to the CIS estimate for 1 sample. (d) Grouping terms 1 and 2 together,where Uc = max πc e(a|c) πc b (a|c). The first te...

work page 2016
[4]

Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features

URL https://arxiv.org/ abs/2210.02401. Josiah Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31,

work page Pith review arXiv
[8]

mlr.press/v119/koh20a.html

URL https://proceedings. mlr.press/v119/koh20a.html. Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine , 24(11):1716—1720, November 2018a. ISSN 1078-8956. doi: 10.1038/ s41591-018-0213-5. URL https://doi....

doi:10.1038/s41591-018-0213-5
[9]

Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling

URL https://arxiv.org/abs/ 1910.06508. Yao Liu, Yannis Flet-Berliac, and Emma Brunskill. Offline policy optimization with eligible actions,

work page Pith review arXiv 1910
[10]

Offline Policy Optimization with Eligible Actions

URL https://arxiv.org/abs/2207.00632. Pedro Madeira, André Carreiro, Alex Gaudio, Luís Rosado, Filipe Soares, and Asim Smailagic. Zebra: Explaining rare cases through outlying interpretable concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3781–3787,

work page Pith review arXiv
[12]

Symbolic Regression for PDEs using Pruned Differentiable Programs

URL https://arxiv.org/abs/2303.07009. Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, volume 1077,

work page Pith review arXiv
[17]

Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction

Brahma S. Pavse and Josiah P. Hanna. Scaling marginalized importance sampling to high-dimensional state-spaces via state abstraction, 2022a. Brahma S. Pavse and Josiah P. Hanna. Scaling marginalized importance sampling to high-dimensional state-spaces via state abstraction, 2022b. URL https://arxiv.org/abs/2212.07486. Achim Peine, Andreas Hallawa, Jan Bic...

work page Pith review arXiv
[18]

Doina Precup, Richard S

doi: 10.1038/s41746-021-00388-6. Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, pp. 759–766, San Francisco, CA, USA,

doi:10.1038/s41746-021-00388-6

Show all 27 references

[20]

Richard S Sutton and Andrew G Barto

doi: 10.1001/jama.2016.0287. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press,

2016
[21]

Philip Thomas and Emma Brunskill

URL https://arxiv.org/abs/2310.17146. Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 2139–2148. PMLR,

arXiv
[22]

Carissa Wu, Sonali Parbhoo, Marton Havasi, and Finale Doshi-Velez

doi: 10.1056/NEJMra1208943. Carissa Wu, Sonali Parbhoo, Marton Havasi, and Finale Doshi-Velez. Learning optimal summaries of clinical time-series with concept bottleneck models. In Machine Learning for Healthcare Conference, pp. 648–672. PMLR,

doi
[25]

D.1.7 V ARIANCE COMPARISON WITH MIS ESTIMATOR Theorem

Thus, K = K1 K2 < 1 and Hence, O( T 2R2 maxU 2T c N ) ∼ O( T 2R2 max(UsK)2T N ) ∼ O( T 2R2 maxUs 2T N )K 2T (3) Thus, the Concept-based MSE bounds are tightened by a factor of K 2T . D.1.7 V ARIANCE COMPARISON WITH MIS ESTIMATOR Theorem. Let ρ be the product of the Importance ...

2020
[26]

We focus on the task of managing acutely hypotensive patients in the ICU

from Phys- ioNet (Goldberger et al., 2000), which records the treatment and progression of ICU patients at the Beth Israel Deaconess Medical Center in Boston, Massachusetts. We focus on the task of managing acutely hypotensive patients in the ICU. Our preprocessing follows the...

2018
[65]

A MAP above 65 indicates that the patient is not experiencing hypotension. 31 0 4 8 12 16 20 X 0 4 8 12 16 20Y Penalty: 0 Wind: (+1,-1) Penalty: -2 Wind: (+0,-1) Penalty: -1 Wind: (+0,-1) Penalty: -1 Wind: (+0,-1) Penalty: -1 Penalty: 0 Penalty: 0 Wind: (+1,-1) Penalty: -2 Win...

2020
[1998]

ISBN 0897919629

Association for Computing Machinery. ISBN 0897919629. doi: 10.1145/276698.276876. URL https://doi.org/10.1145/276698. 276876. Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 652...
[2001]

URL https://doi.org/10

doi: 10.1198/016214501753382327. URL https://doi.org/10. 1198/016214501753382327. PMID: 20019887. Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections,

doi
[2013]

Hierarchical concept discovery models: A concept pyramid scheme

Konstantinos P Panousis, Dino Ienco, and Diego Marcos. Hierarchical concept discovery models: A concept pyramid scheme. arXiv preprint arXiv:2310.02116,

arXiv
[2014]

Beyond concept bottleneck models: How to make black boxes intervenable? arXiv preprint arXiv:2401.13544,

Riˇcards Marcinkevi ˇcs, Sonia Laguna, Moritz Vandenhirtz, and Julia E V ogt. Beyond concept bottleneck models: How to make black boxes intervenable? arXiv preprint arXiv:2401.13544,

arXiv
[2016]

doi: 10.1038/sdata.2016.35

ISSN 2052-4463. doi: 10.1038/sdata.2016.35. URL https://europepmc.org/articles/PMC4878278. Pulkit Katdare, Nan Jiang, and Katherine Rose Driggs-Campbell. Marginalized importance sampling for off-environment policy evaluation. In Jie Tan, Marc Toussaint, and Kourosh Darvish (ed...

2016 doi
[2017]

Simon P Shen, Yecheng Ma, Omer Gottesman, and Finale Doshi-Velez

URL https://arxiv.org/abs/1707.06347. Simon P Shen, Yecheng Ma, Omer Gottesman, and Finale Doshi-Velez. State relevance for off-policy evaluation. In International Conference on Machine Learning, pp. 9537–9546. PMLR,

arXiv
[2018]

Ramtin Keramati, Omer Gottesman, Leo Anthony Celi, Finale Doshi-Velez, and Emma Brun- skill

doi: 10.1038/ s41581-018-0052-0. Ramtin Keramati, Omer Gottesman, Leo Anthony Celi, Finale Doshi-Velez, and Emma Brun- skill. Identification of subgroups with similar benefits in off-policy policy evaluation. CoRR, abs/2111.14272, 2021a. URL https://arxiv.org/abs/2111.14272. R...

arXiv
[2019]

Gendice: Generalized offline estimation of stationary values, 2020a

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values, 2020a. Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values, 2020b. A C ONCEPT DESIDERATA Ex...

2021
[2020]

URL https://arxiv.org/abs/2012. 02149. David M. Bossens and Philip S. Thomas. Low variance off-policy evaluation with state-based importance sampling,

2012
[2021]

S A Murphy, M J van der Laan, J M Robins, and Conduct Problems Prevention Research Group

URL https://arxiv.org/abs/2111.11113. S A Murphy, M J van der Laan, J M Robins, and Conduct Problems Prevention Research Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96 (456):1410–1423,

arXiv
[2022]

Yinlam Chow, Marek Petrik, and Mohammad Ghavamzadeh

doi: 10.1371/journal.pone.0275358. Yinlam Chow, Marek Petrik, and Mohammad Ghavamzadeh. Robust policy optimization with baseline guarantees. arXiv preprint arXiv:1506.04514,

arXiv
[2023]

Promises and pitfalls of black-box concept learning models

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314,

arXiv
[2024]

Markus Böck, Julien Malle, Daniel Pasterk, Hrvoje Kukina, Ramin Hasani, and Clemens Heitzinger

URL https://arxiv.org/abs/2212.03932. Markus Böck, Julien Malle, Daniel Pasterk, Hrvoje Kukina, Ramin Hasani, and Clemens Heitzinger. Superhuman performance on sepsis mimic-iii data by distributional reinforcement learning. PLOS ONE, 17:e0275358, 11

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

(b) The second term is always greater than 0 (c) Applying Bernstein inequality with probability 1- δ

) + vuut ln( 2 δ ) N 3 − N 2 NX i<j (X 2 i − X 2 j )2 (d) Explanation of steps: (a) We begin with the definition of variance. (b) The second term is always greater than 0 (c) Applying Bernstein inequality with probability 1- δ. Xi refers to the CIS estimate for 1 sample. (d) Grouping terms 1 and 2 together,where Uc = max πc e(a|c) πc b (a|c). The first te...

work page 2016

[2] [4]

Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features

URL https://arxiv.org/ abs/2210.02401. Josiah Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31,

work page Pith review arXiv

[3] [8]

mlr.press/v119/koh20a.html

URL https://proceedings. mlr.press/v119/koh20a.html. Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine , 24(11):1716—1720, November 2018a. ISSN 1078-8956. doi: 10.1038/ s41591-018-0213-5. URL https://doi....

doi:10.1038/s41591-018-0213-5

[4] [9]

Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling

URL https://arxiv.org/abs/ 1910.06508. Yao Liu, Yannis Flet-Berliac, and Emma Brunskill. Offline policy optimization with eligible actions,

work page Pith review arXiv 1910

[5] [10]

Offline Policy Optimization with Eligible Actions

URL https://arxiv.org/abs/2207.00632. Pedro Madeira, André Carreiro, Alex Gaudio, Luís Rosado, Filipe Soares, and Asim Smailagic. Zebra: Explaining rare cases through outlying interpretable concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3781–3787,

work page Pith review arXiv

[6] [12]

Symbolic Regression for PDEs using Pruned Differentiable Programs

URL https://arxiv.org/abs/2303.07009. Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, volume 1077,

work page Pith review arXiv

[7] [17]

Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction

Brahma S. Pavse and Josiah P. Hanna. Scaling marginalized importance sampling to high-dimensional state-spaces via state abstraction, 2022a. Brahma S. Pavse and Josiah P. Hanna. Scaling marginalized importance sampling to high-dimensional state-spaces via state abstraction, 2022b. URL https://arxiv.org/abs/2212.07486. Achim Peine, Andreas Hallawa, Jan Bic...

work page Pith review arXiv

[8] [18]

Doina Precup, Richard S

doi: 10.1038/s41746-021-00388-6. Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, pp. 759–766, San Francisco, CA, USA,

doi:10.1038/s41746-021-00388-6