{"id":"275e04f5-7bad-4ef2-84d3-f169763db045","arxiv_id":"2411.19395","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Concept-based importance sampling for off-policy evaluation is introduced, claiming unbiasedness and variance reduction for known concepts and learning concepts with a CBM algorithm when unknown.","lead":"This paper proposes using interpretable concepts, rather than raw states, inside importance-sampling estimators for off-policy evaluation, and claims this reduces variance while remaining unbiased when concepts are known. It also introduces an algorithm that learns such concepts from data and reports large variance reductions on a gridworld and on ICU hypotension data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 5.3's unbiasedness proof assumes data are generated by the concept-based behavior policy π^c_b, while the paper's OPE setup samples from state-based π_b; the cancellation in Appendix D.1.1 therefore does not hold, leaving the central known-concept guarantee unsupported.","rationale":"I read the paper as claiming that known concepts yield unbiased, lower-variance OPE, with Theorem 5.3 as the keystone. The proof's sampling-measure error is the most load-bearing issue: the unbiasedness result depends on a change of measure that the setup does not provide. This is not a disagreement with consensus; it is an internal mismatch between Section 3's data-generating process and Appendix D.1.1's expectation. The empirical bias in Figure 2 corroborates the problem rather than contradicting it. The variance comparison also rests on an assumed covariance inequality rather than a derivation, and the Appendix D.1.3 induction has a sign error, but those are secondary because even a correctly proved conditional variance bound would not fix the missing unbiasedness. I agree with the reader's weakest-assumption identification. The paper does contain an interesting empirical finding—concept-based reweighting can reduce variance, and the learned-concept and intervention framework has practical potential—and the released code is a point in its favor, but the theoretical foundation for the headline claim is not established. The reject verdict stands.","tokens_in":35340,"tokens_out":4435,"duration_ms":41272,"concrete_test":"Re-derive the bias of CIS from the actual sampling distribution: E_{τ∼π_b}[ρ^c_{0:T} ∑ γ^t r_t] = ∑_τ (∏_t π_b(a_t|s_t)) (∏_t π^c_e(a_t|c_t)/π^c_b(a_t|c_t)) ∑ γ^t r_t. Compare this with V_{π_e} = ∑_τ (∏_t π_e(a_t|s_t)) ∑ γ^t r_t on a small MDP where two states share one concept but have different π_b action probabilities—for example, s1 and s2 both map to c, with π_b(a1|s1)=0.9, π_b(a1|s2)=0.1, π^c_b(a1|c)=0.5, and π_e and π^c_e chosen to satisfy Assumption 5.1. If the two expressions differ, Theorem 5.3 is false; if they coincide for all such MDPs, the proof gap is repairable.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing step is Appendix D.1.1. Line (a) sets Bias = |E_{π^c_b}[\\hat V^{CIS}_{π^c_e}] − E_{π^c_e}[\\hat V^{CIS}_{π^c_e}]|, and line (c) writes the trajectory probability as ∏_t π^c_b(a_t|c_t), so line (e)'s cancellation is valid only under π^c_b. But Section 3 defines the batch data as generated by a state-based behavior policy π_b(a|s), and Definitions 4.1–4.2 reweight with π^c_e(a|c)/π^c_b(a|c). Unbiasedness for V_{π_e} requires E_{τ∼π_b}[ρ^c_{0:T} ∑ γ^t r_t] = V_{π_e}; the proof never establishes this. Assumption 5.1 only asserts support, and the Lemma in Appendix D relates expectations of functions of c under state and concept visitation, not trajectory probabilities or action policies. Unless π_b(a|s) = π^c_b(a|ϕ(s)) for every state–action pair—which is not assumed—the telescoping cancellation fails and Theorem 5.3 is not proven. The paper's own Figure 2 reports higher bias for known concepts, even for oracle concepts, consistent with this failure rather than with finite-sample noise. A second compounding flaw is the variance-ratio induction in Appendix D.1.3: substituting C ≤ √(SC′) into D′ = (SC′−C)X²/(C′Y) gives a lower bound, not the upper bound claimed, so the conclusion D′ ≤ 0 has the wrong sign for the required D′ ≥ 0.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes replacing the state in importance-sampling (IS) ratios for off-policy evaluation (OPE) with human-interpretable concepts, defining concept-based IS (CIS) and concept-based per-decision IS (CPDIS) estimators. For known concepts it claims unbiasedness (Theorem 5.3) and lower variance relative to ordinary IS, PDIS, and MIS under covariance conditions (Theorems 5.4 and 5.5), and it develops an end-to-end algorithm (Algorithm 1) that learns parameterized concepts via a concept bottleneck model with a loss that includes the OPE estimator's variance. Experiments on WindyGridworld and MIMIC-III report large variance and MSE reductions for concept-based estimators, while also reporting higher bias even for known oracle concepts, and an intervention analysis is used to interpret and adjust learned concepts.","tokens_in":35753,"tokens_out":4982,"duration_ms":46434,"significance":"If the theoretical claims were correct, the paper would make a useful contribution: grouping states into interpretable concepts could reduce OPE variance while providing interpretability and targeted interventions. The paper also has strengths: it provides a code link, detailed appendices, a real-world medical dataset, an ablation with imperfect concepts, and a comparison with state abstractions. However, the central unbiasedness theorem is proved under a different sampling model than the one defined in the paper, and the reported experiments directly contradict it. The variance-reduction proof contains a sign error, and the covariance conditions in the variance theorems largely restate the desired conclusion. As a result, the main theoretical and empirical claims are not currently established.","major_comments":[{"comment":"The proof of Theorem 5.3 computes the bias under the concept-based behavior policy π^c_b: line (a) of Appendix D.1.1 writes E_{π^c_b}[V-hat^{CIS}], and line (c) writes the trajectory probability as ∏_t π^c_b(a_t|c_t), so the cancellation in step (e) is valid only when trajectories are generated by π^c_b. But Section 3 defines the batch data as generated by the state-based policy π_b(a|s), and Assumption 5.1 only asserts support/completeness; it does not imply π_b(a|s) = π^c_b(a|ϕ(s)) for all (s,a). Unbiasedness for V_{π_e} requires E_{τ∼π_b}[ρ^c_{0:T} ∑ γ^t r_t] = V_{π_e}, which the proof never establishes. This is load-bearing because Theorem 5.3 is the paper's headline guarantee. The contradiction is visible in Figure 2, where known-concept estimators, including oracle concepts, show nonzero bias; Section 5.3 attributes this to poor coverage, but that explanation is not consistent with the claimed proof under Assumption 5.1.","section":"Section 3, Definitions 4.1-4.2, Appendix D.1.1, Theorem 5.3"},{"comment":"The induction proof that the concept-based IS ratio has lower variance than the state-based IS ratio has a sign error. The goal is to show D' ≥ 0, but after substituting C ≤ √(SC') into D' = (SC' − C)X²/(C'Y), the text derives D' ≤ 0 and then states that the induction hypothesis holds. Moreover, the claim that SC' − √(SC') is never negative is false when 0 ≤ SC' < 1. Thus the variance-ratio comparison is not proven, and this invalidates the ratio-level inequality used in the variance comparisons of Theorems 5.4 and 5.5.","section":"Appendix D.1.3, Equations (h)-(m)"},{"comment":"The covariance assumptions in Theorems 5.4 and 5.5 are effectively restatements of part of the desired variance inequality. The theorem states that if Cov(ρ^c_{0:t}r_t, ρ^c_{0:k}r_k) ≤ Cov(ρ_{0:t}r_t, ρ_{0:k}r_k), then the concept-based estimator has lower variance; but this covariance condition is not derived from the concept mapping or from Assumptions 5.1-5.2. Without a structural argument showing that grouping states into concepts implies the condition, the theorem is a conditional identity rather than a substantive variance-reduction guarantee. When rewards are fixed to 1, the condition reduces exactly to the ratio-variance comparison in Appendix D.1.3, whose proof fails.","section":"Theorems 5.4 and 5.5"},{"comment":"The unknown-concept section is a central claimed contribution, but Algorithm 1 uses the OPE estimator's variance on the training trajectories as a training loss (Line 12), then reports results on a held-out split. This is not circular in the narrow sense of evaluating on the same data, but the paper provides no analysis of how this variance-minimizing training affects the bias-variance trade-off, and Theorem 6.1 concedes that unknown-concept estimators are biased. Given that the experiments show higher bias for learned concepts, the claim that the algorithm produces better OPE estimators needs a more careful treatment of the bias introduced by the concept-learning objective rather than only an empirical variance reduction.","section":"Algorithm 1, Line 12; Section 6"}],"minor_comments":[{"comment":"Assumption 5.2 states the same inequality twice: both conditions read |π^c_e(a|c) − π_e(a|s)| < β. The second inequality should presumably involve π^c_b and π_b.","section":"Assumption 5.2"},{"comment":"The concept-based behavior policy π^c_b(a|c) and evaluation policy π^c_e(a|c) are used in the estimators but are never formally defined in Section 4; the paper should state how they are derived from π_b, π_e, and the concept mapping ϕ.","section":"Definitions 4.1 and 4.2"},{"comment":"The notation K is used inconsistently: Theorem 5.6 describes K as the ratio of the cardinality of the concept space to the state space, whereas Appendix D.1.6 defines K = K1/K2 through counts of states mapping to a concept under π_e and π_b. A single consistent definition is needed.","section":"Theorem 5.6 and Appendix D.1.6"},{"comment":"The text says known-concept estimators are 'unbiased' in theory but reports higher bias in practice, even for oracle concepts. This discrepancy is not resolved in the paper and should be discussed explicitly, especially because it bears on the validity of Theorem 5.3.","section":"Section 5.3, Figure 2"},{"comment":"The three-stage training strategy is described in prose only; providing a precise schedule or pseudocode for when the OPE-metric loss is activated would improve reproducibility.","section":"Appendix G.1"}],"recommendation":"reject","confidential_remarks":"The gap in Appendix D.1.1 is not a local fix: the estimators are defined through concept-based policies while the data distribution is state-based, so repairing the unbiasedness claim would require either changing the data-generating model, adding a strong assumption that π_b equals π^c_b, or substantially weakening the paper's central claim. The reported bias for oracle known concepts in Figure 2 further undermines Theorem 5.3. I therefore recommend rejection, although the paper's topic and experimental effort are not without merit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: the core idea—doing OPE through interpretable concepts and learning those concepts with a CBM—is genuinely worth discussing, but the main theoretical guarantee is not proven for the setting the paper defines. Section 3 says the batch data come from a state-based behavior policy π_b(a|s), yet the proof of Theorem 5.3 in Appendix D.1.1 computes expectations under the concept-based policy π^c_b(a|c) and cancels ratios that only telescope if the data distribution factors through concepts. Assumption 5.1 does not bridge that gap. Unsurprisingly, Figure 2 shows bias for known concepts even with oracle concepts; that is not finite-sample noise, it is the theorem failing against the paper's own setup.\n\nWhat is genuinely new: the end-to-end CBM concept learner with an OPE-variance loss (Algorithm 1) and the intervention framework. The known-concept CIS estimator, however, is mathematically the same as abstraction-based importance sampling (Pavse and Hanna, 2022b), which the paper cites but does not flag as a direct overlap. The experiments are reasonably extensive, the code is available, and the empirical variance reductions beyond bias look real. The imperfect-concepts ablation is a nice touch.\n\nThe soft spots beyond the bias proof are serious. The variance comparison in Appendix D.1.3 has a sign error in the induction: substituting C ≤ sqrt(SC′) gives a lower bound on D′, not an upper bound, and the proof concludes D′ ≤ 0 where D′ ≥ 0 was needed. Theorems 5.4 and 5.5 put the desired covariance inequality as an assumption, so they are conditional statements rather than derived guarantees. For unknown concepts, Algorithm 1 directly optimizes the variance it later reports; the bias is acknowledged but not quantified, and the Appendix E bounds are loose.\n\nBottom line: the framing and the empirical story are intriguing, but the load-bearing theory is broken. I would still send this to a knowledgeable referee rather than desk-reject—there is enough of a kernel that a major revision could produce something solid. As it stands, the central claims should not be cited.\n\nIf you work on OPE, engage with it; otherwise skim the experiments and the intervention section.","headline":"Concept-based OPE is a promising framing, but the central unbiasedness proof assumes the wrong behavior policy and the variance induction has a sign error, so the headline guarantees do not hold as written.","tokens_in":36272,"tokens_out":6285,"would_cite":false,"duration_ms":57553,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Concept-based importance sampling preserves unbiasedness and cuts variance in off-policy evaluation.","keywords":["off-policy evaluation","importance sampling","concept bottleneck models","interpretable concepts","variance reduction","reinforcement learning","concept interventions"],"falsifier":"Run a synthetic MDP with a known evaluation policy and a behavior policy that depends on a state feature not captured by any concept map $\\phi$, then compute the CIS estimator with oracle concepts on an arbitrarily large batch; if the estimate does not converge to the true $V_{\\pi_e}$, the unbiasedness theorem's data-generation premise is violated.","tokens_in":35065,"feed_emoji":"🧩","tokens_out":9650,"duration_ms":82889,"temperature":0.7,"pith_summary":"The paper proposes replacing the state in an importance-sampling ratio with an interpretable concept—a coarse grouping of states that a human can understand—and claims this makes off-policy evaluation more reliable and more explainable. For known concepts, it proves that the resulting CIS and CPDIS estimators stay unbiased under the completeness assumption, and that their variance is no larger than that of standard IS, PDIS, and marginalized-IS estimators when a per-timestep covariance condition holds. For unknown concepts, it learns concepts end-to-end through a concept bottleneck model whose loss includes the OPE variance itself; experiments in a gridworld and on ICU patient data show variance reductions of one to three orders of magnitude at the cost of some bias. The paper also shows that intervening on high-variance learned concepts, guided by domain knowledge, can further reduce bias or variance.","feed_headline":"Concept grouping cuts off-policy variance without losing unbiasedness","feed_subtitle":"Known and learned concept-based estimators beat state-based importance sampling, and concepts allow targeted fixes.","key_machinery":"The central object is the concept map $\\phi: S \\to C$ that sends each state to an interpretable concept vector $c_t$, together with the concept-conditioned policies $\\pi^c_e$ and $\\pi^c_b$ used in the importance ratio $\\rho^c_{0:t}=\\prod_{t'=0}^t \\pi^c_e(a_{t'}|c_{t'})/\\pi^c_b(a_{t'}|c_{t'})$. The variance comparison is carried by the covariance condition $\\mathrm{Cov}(\\rho^c_{0:t}r_t, \\rho^c_{0:k}r_k) \\le \\mathrm{Cov}(\\rho_{0:t}r_t, \\rho_{0:k}r_k)$, and the proofs adapt a state-abstraction lemma showing the per-concept variance difference is nonpositive. For unknown concepts, the machinery is a concept bottleneck model whose bottleneck output is trained with a combined loss: next-state prediction, L1 interpretability, cosine diversity, a policy-closeness term enforcing the $\\beta$ deviation bound, and the OPE variance itself.","core_discovery":"The paper's central claim is that importance ratios conditioned on a concept bottleneck, $\\pi^c_e(a_t|c_t)/\\pi^c_b(a_t|c_t)$ instead of $\\pi_e(a_t|s_t)/\\pi_b(a_t|s_t)$, preserve the unbiasedness of importance sampling while lowering variance whenever concepts group states with similar action distributions. Theorem 5.3 states unbiasedness for both CIS and CPDIS under the completeness assumption; Theorem 5.4 states the variance comparison to IS and PDIS under a covariance inequality; Theorem 5.5 states the comparison to marginalized importance sampling; and Theorem 5.6 says the Cramer-Rao MSE bound tightens by $K^{2T}$, where $K$ is the ratio of concept-cardinality to state-cardinality. For learned concepts, Theorem 6.1 concedes finite bias because the change of measure from $\\pi_b$ to $\\pi^c_b$ is no longer available, while the variance bounds survive under the same covariance condition. The unbiasedness proof in Appendix D.1.1 takes its first expectation under the concept-conditioned behavior policy $\\pi^c_b$, so the telescope result is conditional on the data actually being generated by a concept-conditioned policy.","pith_inferences":["A direct test of the unbiasedness claim is to check whether real logged data satisfy the concept-conditioned behavior-policy premise; if they do not, the observed bias in Figure 2 is the expected price, and practitioners should reweight or relabel data so that it holds.","The covariance condition in Theorems 5.4 and 5.5 can be estimated from logged trajectories; a validation study on hospital data would show how often concept groupings actually satisfy it, which is the empirical question the paper leaves open.","Because the learned-concept loss minimizes variance but not bias (the true on-policy value is unavailable), a natural extension is to add a model-based or doubly robust bias term to the loss, which could recover unbiasedness without sacrificing interpretability.","The $K^{2T}$ factor points to a trade-off: coarser concepts shrink variance but may violate the policy-closeness assumption, so concept learning should monitor the deviation bound alongside the OPE metric."],"forward_implications":["If the theorems hold, any OPE pipeline can replace state-conditioned importance ratios with concept-conditioned ones and, under the completeness and covariance conditions, inherit unbiasedness with lower variance.","The $K^{2T}$ tightening of the MSE bound makes concept coarseness a design lever: coarser concept spaces improve worst-case sample complexity exponentially in the horizon.","The learned-concept algorithm offers a variance-aware concept discovery procedure that, in the paper's experiments, beats human-defined concepts in both WindyGridworld and MIMIC-III while remaining interpretable.","Concept-based interventions give practitioners a concrete mechanism: replace or remove a high-variance concept and re-run the estimator, instead of treating OPE as a black box.","The imperfect-concept ablation shows concept quality is load-bearing; concepts that violate the desiderata can perform worse than state-based IS, so the desiderata function as necessary conditions."],"supporting_citations":[{"why":"Defines the per-decision importance sampling estimator whose unbiasedness and variance structure the concept-based estimators adapt.","marker":"Precup et al. (2000)"},{"why":"Introduces concept bottleneck models, the interpretable-concept machinery used to learn concepts when they are unknown.","marker":"Koh et al. (2020)"},{"why":"Supplies the covariance assumptions and doubly robust OPE framing that the paper adopts for its variance comparison theorems.","marker":"Jiang & Li (2016)"},{"why":"Provides the state-abstraction variance-reduction lemma and proof template the paper generalizes from state abstractions to concepts.","marker":"Pavse & Hanna (2022b)"},{"why":"Provides conditional importance-sampling identities used to compare concept-based estimators against marginalized importance sampling.","marker":"Liu et al. (2020)"},{"why":"Establishes the subgroup-OPE setting that concept-based estimators extend with interpretable concepts rather than regression-tree subgroups.","marker":"Keramati et al. (2021b)"}],"fun_headline_variants":["Concept bottlenecks slash off-policy variance without bias","Group states by concepts to reduce OPE variance","Known concepts give unbiased OPE with lower variance","Learned concepts trade tiny bias for big variance cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the batch data were collected by a policy that chooses actions using only the concept label, not the full state; only then does the concept-based importance weight telescope to give unbiasedness, and the proof in Appendix D.1.1 assumes exactly this in its first expectation.","fun_headline_variants_meta":{"raw":{"variants":["Concept bottlenecks slash off-policy variance without bias","Group states by concepts to reduce OPE variance","Known concepts give unbiased OPE with lower variance","Learned concepts trade tiny bias for big variance cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2902,"prompt_tokens":959,"completion_tokens":1943,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1883}},"tokens_in":575,"tokens_out":1943,"duration_ms":13428,"temperature":1.0,"reasoning_tokens":1883,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:13:41.157524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a synthetic MDP with a known evaluation policy and a behavior policy that depends on a state feature not captured by any concept map $\\phi$, then compute the CIS estimator with oracle concepts on an arbitrarily large batch; if the estimate does not converge to the true $V_{\\pi_e}$, the unbiasedness theorem's data-generation premise is violated.","supporting_citations":[],"review_version":1}