{"id":"52b9c70f-5b46-4c19-83ff-ecdd3f446f0b","arxiv_id":"2412.08052","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using imperfect counterfactual annotations only in the reward model part of a doubly robust estimator is the theoretically and empirically safest way to incorporate them.","lead":"Researchers propose three ways to add expert-annotated counterfactual samples to doubly robust off-policy evaluation and prove that putting imperfect annotations only in the reward model component keeps the estimator unbiased. On simulated healthcare bandit tasks, that strategy is most robust when both the reward model and the annotations are imperfect.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sepsis experiments violate Assumption 4 (common support), so Proposition 1's unbiasedness of DM+-IS does not apply in that environment; the empirical claim 'across all datasets' is not theoretically grounded.","rationale":"The reader's weakest assumption concerns dataset independence for the reward-model fit; that is a practical caveat, and the paper states the two-dataset assumption explicitly in Section 4. The coverage violation is more serious because the paper's own sepsis experiments violate an assumption of the central proposition, and the conclusion is drawn 'across all datasets,' including sepsis. This directly threatens the empirical validation of the main recommendation. The variance-theorem algebraic error noted by the reader is real but secondary: the central ranking comes from unbiasedness (Proposition 1) and bias (Theorem 2), not from Theorem 3. If the sepsis results are re-run under full coverage and DM+-IS no longer dominates, the paper's headline practical guide loses its primary empirical support. Thus the verdict remains CONDITIONAL, contingent on correcting the experiment-theory mismatch. I disagree with the reader's identification of the weakest assumption because the coverage issue appears more load-bearing and is internally inconsistent with the paper's own experiments.","tokens_in":33927,"tokens_out":15312,"duration_ms":149510,"concrete_test":"Re-run the Sepsis experiments (Figure 3c) using only behavior policies that satisfy Assumption 4, e.g., πb4 and πb5 from Appendix B.3, which have no zero entries, and recompute the RMSE ranking across annotation-bias levels. If DM+-IS is still the best or tied best in this full-support subset, the empirical claim survives; if another estimator (e.g., DM-IS+ or DM+-IS+) wins, the cross-environment conclusion is an artifact of coverage-violating policies outside Proposition 1's scope. As a secondary check, numerically evaluate the bias term Σ_{a:πb=0} πe(a|s)(R̄(s,a) − E[ R̂+(s,a) ]) for the original πb1/πe pair; a nonzero value confirms the theoretical mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4's Proposition 1 establishes unbiasedness of DM+-IS only under Assumption 4 (πe(a|s)>0 ⇒ πb(a|s)>0). The proof in Appendix H.1.1 cancels the reward-model term by converting the IS correction to an expectation over πe: E_{a∼πb}[ρ(a) f(a)] = E_{a∼πe}[f(a)]. This identity requires full support. In the Sepsis environment (Appendix B.3), the listed behavior policies have zero probability on actions that πe supports: e.g., πb1 = [0.1,0.1,0.4,0.3,0.1,0,0,0] while πe = [0.3,0.2,0,0,0.2,0.1,0.1,0.1], so actions 6,7,8 are unsupported. For such (s,a), ρ is undefined or absent, the cancellation fails, and the estimator carries an unanalyzed bias of roughly Σ_{a:πb=0} πe(a|s)(R̄(s,a) − E[ R̂+(s,a) ]). The paper still averages RMSE over these policies (Figure 3c) and concludes DM+-IS is 'most robust across all datasets,' a regime not covered by the theory. This is an internal inconsistency between the theorem's assumptions and the experiment, and it matters because the motivating scenario for counterfactual annotations is precisely insufficient coverage.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies off-policy evaluation (OPE) in contextual bandits when counterfactual annotations are available but possibly imperfect. It proposes three doubly robust inspired estimators that incorporate annotations in different places: DM+-IS uses annotations only in the reward model, DM-IS+ uses them only in the importance sampling part, and DM+-IS+ uses them in both. The main theoretical result is Proposition 1, which states that under biased annotations and common support, DM+-IS is unbiased, while Theorem 2 shows that DM-IS+ and DM+-IS+ carry an annotation-bias term. Theorem 3 gives a variance decomposition for DM+-IS. Experiments on a two-context bandit, Heartsteps, and a Sepsis treatment simulator compare the estimators under misspecified reward models and imperfect annotations, leading the authors to recommend DM+-IS as the most robust choice. The appendix contains proofs, additional experiments, and a discussion of weighted versus unweighted reward models.","tokens_in":34217,"tokens_out":11717,"duration_ms":116985,"significance":"If the main claims hold, the paper provides clear practical guidance: when counterfactual annotations are imperfect, they should be used only in the direct-method component of a doubly robust estimator. Proposition 1 is a clean and useful result, and the paper is a rare OPE contribution that both proves a property and ships code and detailed empirical comparisons. However, the proof of Theorem 3 contains an algebraic error that invalidates the stated variance formula, and the Sepsis experiments violate Assumption 4 on which Proposition 1 depends. These issues affect the formal and empirical support for the 'most robust across all datasets' claim, so the manuscript needs revision before the recommendations can be accepted as stated.","major_comments":[{"comment":"The displayed derivation does not establish the stated variance formula. After the unbiased-DM substitution, the derivation reaches an expression of the form E_s[ Σ_a πe(a|s)^2 (1/πb(a|s) - 1) V_{D̂R}[R̂+(s,a)] ]. The next displayed line rewrites this as E_s E_{a∼πb}[(ρs(a)^2 - 1/πb(a|s)) Δ_{R̂+}(s,a)], which equals Σ_a (πe(a|s)^2/πb(a|s) - 1) Δ_{R̂+}(s,a), not the previous expression; the factor πe(a|s)^2 on the second term has been replaced by 1. The valid simplification would keep E_s[ E_{a∼πb}[ρs(a)^2 Δ_{R̂+}(s,a)] - Σ_a πe(a|s)^2 Δ_{R̂+}(s,a) ]. Since Theorem 3 is the variance result listed in Table 3 and is invoked in Section 6.1 to interpret the RMSE behavior of DM+-IS, the theorem must be corrected or explicitly qualified. The same passage also contains an apparent algebraic slip in the biased-DM case, where a term 2vπe(s)^2 R̃(s,πe) appears instead of 2vπe(s)R̃(s,πe).","section":"Appendix H.1.2 (Theorem 3)"},{"comment":"The Sepsis experiments include behavior policies that violate Assumption 4, which is required by Proposition 1. For example, πb1 = [0.1,0.1,0.4,0.3,0.1,0,0,0] and πb2 = [0.1,0.1,0.4,0.2,0.1,0.1,0,0] place zero mass on actions that the target policy πe = [0.3,0.2,0,0,0.2,0.1,0.1,0.1] supports. The proof of Proposition 1 uses the identity E_{a∼πb}[ρ(a)f(a)] = E_{a∼πe}[f(a)], which fails when πb does not cover πe. For unsupported actions, the estimator carries an additional, unmodeled bias of order Σ_{a:πb(a|s)=0} πe(a|s)(R̄(s,a) - E[R̂+(s,a)]), which is not analyzed. Nevertheless, Figure 3c averages RMSE over these behavior policies and Section 6.2 concludes that DM+-IS is most robust 'across all datasets.' Please either restrict the Sepsis evaluation to behavior policies satisfying Assumption 4 or extend the theoretical analysis to partial coverage and report the corresponding bias term.","section":"Appendix B.3 and Section 6.2"}],"minor_comments":[{"comment":"The displayed variance formula has a bracket mismatch and an unclosed expectation expression; please correct the typesetting so that the formula is parseable.","section":"Section 4, Theorem 3 display"},{"comment":"Propositions 12 and 13 are stated under Assumptions 1 and 5, but their proofs use the standard IS ratio and rely on Assumption 4 (common support without annotations). Please align the stated assumptions with the proofs.","section":"Appendix H.1.1, Propositions 12 and 13"},{"comment":"The reported total compute time differs between the acknowledgements (approximately 30 hours) and Appendix C (approximately 100 hours); please reconcile these numbers.","section":"Appendix C and Acknowledgements"},{"comment":"The corollary title lists DM+-IS, but the statement and proof concern DM-IS+ and DM+-IS+; please correct the title.","section":"Appendix I, Corollary 18"},{"comment":"Please clarify whether the dataset used to fit R̂ and R̂+ is independent of the dataset used to form the OPE estimate, as required by Section 4; the phrase 'the same dataset size and distribution is used to learn a reward function estimate' is ambiguous.","section":"Appendix B.1"},{"comment":"In the paragraph on unknown reward model and annotation quality, 'annotation equality' should read 'annotation quality.'","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The Theorem 3 issue is more than a typographical slip: the stated variance formula is not derived from the preceding lines. This should be corrected or the variance claim should be downgraded. The Sepsis coverage violation is also important because it directly affects the 'across all datasets' empirical claim. That said, Proposition 1 is sound and the central practical recommendation may survive after these fixes, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on CANDOR. The practical claim is worth taking seriously: under a misspecified reward model and biased counterfactual annotations, you should put the annotations in the DM part of a DR estimator, not the IS part. Proposition 1 and Theorem 2 are clean — the DR cancellation works, and the bias term for the IS+ variants (Eq. 4) matches what you'd expect. The three-estimator family is a genuine extension of the authors' own IS+ work, and the self-citation is not a problem; the unbiasedness result stands on the DR cancellation, not on IS+ being correct. The experiments are decent: three environments, well-specified vs misspecified reward models, annotation bias and variance swept, bootstrapped errors. The Figure 4 lookup table is a useful distillation for practitioners.\n\nNow the soft spots, in order of importance.\n\nFirst, Theorem 3 does not check out. In the appendix proof, going from sum_a pi_e(a)^2 (1/pi_b(a)) V[R_hat+] minus sum_a pi_e(a)^2 V[R_hat+] to the stated final term E_{a~pi_b}[(rho^2 - 1/pi_b)Delta] requires sum_a pi_e(a)^2 V = sum_a V, which fails in general. The variance decomposition as stated is not established. This does not sink the main recommendation, which rides on the bias results, but the theorem needs to be fixed or weakened.\n\nSecond, the sepsis experiments violate Assumption 4 for three of the six behavior policies. pi_b1 through pi_b3 put zero mass on actions that pi_e supports, so rho is undefined, the IS correction does not fire, and DM+-IS carries an unanalyzed bias of sum_{a:pi_b=0} pi_e(a)(E[R_hat+(a)] - R_bar(a)). The paper averages these runs into the across-all-datasets claim without noting that the regime falls outside the theorem. I checked the appendix; the stress-test concern holds. This is the exact regime — insufficient coverage — that motivates counterfactual annotations, so the gap matters.\n\nThird, smaller items: the abstract claims real-world EHR data, but all three environments are simulators; that sentence should be corrected. The theory assumes separate datasets for the OPE estimate and the reward model fit; the single-dataset regime is not analyzed, which is fine as a stated assumption but should be flagged as a practical gap. Code is promised on acceptance, not shipped.\n\nNone of this kills the paper. The bias analysis and practical guidance are solid, and I'd trust the empirical ranking more than the variance theorem. It deserves a serious referee; expect heavy revision, mainly on Theorem 3 and the coverage gap. I'd cite it for the DM+-IS recommendation, and I'd take it to a reading group to argue about the coverage issue.","headline":"The bias analysis and practical recommendation are solid and new; the variance theorem has a real algebra error and the sepsis experiments run outside the theory's coverage assumption.","tokens_in":34767,"tokens_out":7803,"would_cite":true,"duration_ms":70620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that imperfect counterfactual annotations should enter a doubly robust estimator only through its reward model: that estimator stays unbiased under biased annotations, and it performs best when the reward model is…","keywords":["off-policy evaluation","contextual bandits","counterfactual annotations","doubly robust estimation","importance sampling","direct method","annotation bias","healthcare decision-making"],"falsifier":"Run the DM+-IS estimator in a two-dataset regime where Assumptions 2 and 4 provably hold, with annotations biased by a known amount $\\epsilon_G$; Proposition 1 predicts the estimate's bias is exactly zero for every $\\epsilon_G$, so a statistically significant non-zero bias would refute it. A second practical check is to repeat the experiment in the single-dataset regime (reward model and OPE estimate on the same behavior data): if a systematic bias appears that grows with $\\epsilon_G$, the two-dataset independence assumption is load-bearing and the guarantee does not transfer to that regime.","tokens_in":33720,"feed_emoji":"🩺","tokens_out":18174,"duration_ms":145383,"temperature":0.7,"pith_summary":"Off-policy evaluation estimates how a new decision policy would perform before it is deployed, using only data logged under a different behavior policy, and its accuracy is limited by the coverage of that data. Prior work augmented the logged data with expert counterfactual annotations, predicted rewards for actions that were not taken, but those annotations are often imperfect and can make estimates worse than using none at all. This paper asks where inside a doubly robust estimator such annotations should be placed, and proves an asymmetry: when annotations are biased, the estimator that uses them only to fit the reward model (DM+-IS) remains unbiased ($E[\\hat{V}^{\\mathrm{DM}+\\text{-IS}}] = v(\\pi_e)$), while estimators that also feed annotations into the importance-sampling term inherit an explicit bias. The paper argues this is the realistic operating regime, since reward models are typically misspecified and annotations imperfect, and in that regime DM+-IS has the lowest estimation error across three simulated environments including a sepsis treatment setting. If the paper is right, practitioners get a simple default for high-stakes deployments: when annotation and model quality are uncertain, use imperfect annotations only in the reward model.","feed_headline":"Put imperfect annotations only in the reward model","feed_subtitle":"Putting biased labels only in the reward-model term keeps OPE unbiased, and experiments back it as the safest default.","key_machinery":"The central mechanism is the placement of the counterfactual-annotated dataset $D_+$ inside the doubly robust decomposition $\\hat{V} = (1/N)\\sum_i [\\hat{R}(s_i,\\pi_e) + \\rho_{s_i}(a_i)(r_i - \\hat{R}(s_i,a_i))]$. The three estimators are the three nontrivial placements: DM+-IS fits the reward model $\\hat{R}_+$ on $D_+$ and keeps the IS correction on factual rewards; DM-IS+ keeps the standard reward model and pushes annotations into the IS term through the augmented behavior policy $\\pi_b^+(a|s) = \\bar{W}(a|s,a)\\pi_b(a|s) + \\sum_{\\tilde{a}\\neq a} \\bar{W}(a|s,\\tilde{a})\\pi_b(\\tilde{a}|s)$, with per-sample weights that sum to one; DM+-IS+ does both. The augmented policy and weights preserve the factual context distribution when annotations are appended, and the naive alternative that skips them distorts the context distribution and is arbitrarily biased (Appendix D). The load-bearing identity is the telescoping in the proof of Proposition 1: $E_{a\\sim\\pi_b}[\\rho_s(a)(r - \\hat{R}_+(s,a))] = E_{a\\sim\\pi_e}[\\bar{R}(s,a) - \\hat{R}_+(s,a)]$, which exactly cancels the $E[\\hat{R}_+(s,\\pi_e)]$ term and leaves $v(\\pi_e)$ with no dependence on the annotation bias $\\epsilon_G$.","core_discovery":"Under Assumption 2 (annotations with expectation $\\bar{R}(s,a) + \\epsilon_G(s,a)$) and common support between behavior and target policies, Proposition 1 establishes that $E[\\hat{V}^{\\mathrm{DM}+\\text{-IS}}] = v(\\pi_e)$: the estimator that fits its reward model on the annotation-augmented dataset and applies standard importance sampling to factual rewards is unbiased, no matter how biased the annotations are. Theorem 2 shows the two estimators that route annotations through the augmented importance-sampling term, DM-IS+ and DM+-IS+, both carry the same bias term, $E_{s\\sim d_0, a\\sim\\pi_e}[(1 - \\bar{W}(a|s,a)\\pi_b(a|s)/\\pi_b^+(a|s))\\epsilon_G(s,a)]$, which is the bias term that already affected the earlier IS+ estimator. The mechanism is a cancellation: the reward model's annotation-induced bias is exactly offset by the inverse-propensity correction $\\rho(r_i - \\hat{R}_+(s_i,a_i))$ evaluated on factual rewards, which remain unbiased. Empirically, across a two-context bandit, a Heartsteps step-count environment, and a sepsis treatment simulator with misspecified reward models, DM+-IS achieves the lowest RMSE across all tested magnitudes of annotation bias, and the paper concludes that any further use of imperfect annotations beyond the reward model can worsen the estimate. A lookup-table guide and a sensitivity analysis in the sepsis setting support the recommendation that DM+-IS is the safe default when annotation and reward-model quality are unknown.","pith_inferences":["A general design rule suggested by Proposition 1, though not stated in the paper: in a doubly robust estimator, imperfect auxiliary data should enter through the model term that the unbiased factual correction can cancel, never through the reweighted term that inherits the auxiliary data's bias.","The two-dataset assumption implies a practical requirement the paper leaves implicit: practitioners need two separate behavior datasets or sample splitting to realize the unbiasedness guarantee, since the single-dataset regime is not analyzed.","A testable extension the paper does not run: with a fixed annotation budget, the theory suggests annotations should be spent where they most improve the reward model (under-covered contexts and actions), because placement does not affect bias, only the quality of the reward model."],"forward_implications":["When counterfactual annotations are available, they should be used to fit the reward model only; feeding them into the importance-sampling term adds a bias proportional to the annotation bias.","If annotation quality and reward-model misspecification are unknown, the empirical comparison supports DM+-IS as the default estimator, with sepsis-setting estimates within a small margin of the best method across all annotation qualities.","The naive strategy of plugging the annotated dataset directly into a standard doubly robust estimator is unsafe because it silently changes the context distribution and can be arbitrarily biased.","With perfect annotations and equal sample weights, IS+, DM-IS+, and DM+-IS+ coincide, so the choice among those three matters only when annotations are imperfect or weights are unequal.","Under a misspecified reward model, pure DM and DM+ degrade while all doubly robust variants keep the protection of their IS term, and DM+-IS gets that protection without the annotation-bias cost of the IS+ variants."],"supporting_citations":[{"why":"Supplies the IS+ estimator, its sample weights, and the augmented behavior policy $\\pi_b^+$ that the paper's three estimators build on, and the bias term that re-appears in Theorem 2.","marker":"Tang and Wiens [2023]"},{"why":"Defines the doubly robust estimator and its unbiasedness and variance properties that the paper extends to counterfactual-annotated datasets.","marker":"Dudik et al. [2011]"},{"why":"Provides the importance-sampling estimator with inverse propensity scores that serves as the IS component of the proposed estimators.","marker":"Precup et al. [2000]"},{"why":"The sepsis treatment simulator, adapted to the contextual bandit setting, that anchors the most realistic empirical evaluation.","marker":"Oberst and Sontag [2019]"},{"why":"The Heartsteps step-count simulator used as the second of the three evaluation environments.","marker":"Mandyam et al. [2024]"},{"why":"Motivates the misspecified reward-model condition (partial observability in the spirit of ModelFail) used in the two-context bandit experiments.","marker":"Thomas and Brunskill [2016]"}],"fun_headline_variants":["Imperfect annotations? Use them only in the reward model","Safe OPE: keep biased labels out of importance sampling","Why annotation bias cancels in OPE's reward model","CANDOR: counterfactual annotations only in DM for unbiased OPE","Best OPE strategy: route annotations solely through reward model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The unbiasedness proof assumes the reward model $\\hat{R}_+$ is fitted on a behavior dataset separate from the one used to compute the OPE estimate; if a practitioner fits and evaluates on the same data, the cancellation that removes the annotation bias can break, and the paper does not analyze that single-dataset regime.","fun_headline_variants_meta":{"raw":{"variants":["Imperfect annotations? Use them only in the reward model","Safe OPE: keep biased labels out of importance sampling","Why annotation bias cancels in OPE's reward model","CANDOR: counterfactual annotations only in DM for unbiased OPE","Best OPE strategy: route annotations solely through reward model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1403,"prompt_tokens":1096,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":712,"tokens_out":307,"duration_ms":3561,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:17:11.302230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the DM+-IS estimator in a two-dataset regime where Assumptions 2 and 4 provably hold, with annotations biased by a known amount $\\epsilon_G$; Proposition 1 predicts the estimate's bias is exactly zero for every $\\epsilon_G$, so a statistically significant non-zero bias would refute it. A second practical check is to repeat the experiment in the single-dataset regime (reward model and OPE estimate on the same behavior data): if a systematic bias appears that grows with $\\epsilon_G$, the two-dataset independence assumption is load-bearing and the guarantee does not transfer to that regime.","supporting_citations":[],"review_version":1}