{"id":"dd37585a-98d3-4f2a-8408-122fa6d37003","arxiv_id":"2608.12489","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Offline evaluation of deterministic top-k allocation is trustworthy only under logger-target action alignment and credible propensities; nuisance-only cross-fitting worsens the optimizer's curse, and propensity-estimation error can invert the overlap screen.","lead":"Organizations often test a targeting rule on old logged data before deploying it. This paper benchmarks when that offline test can be trusted and finds that the biggest dangers are logger-rule mismatch, optimizer bias that cross-fitting does not fix, and errors in estimating the logging policy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3's 'nuisance-only cross-fitting is counterproductive' is stated broadly, but the exact identity relies on known propensities; with estimated propensities and different learners the sign is untested.","rationale":"The paper is unusually disciplined: it separates exact-value from HT-reference evidence, reports negative results, and explicitly flags the sign of the RQ3 covariance as empirical and the seven-of-eight shared-covariate limitation. The concern is therefore not an internal inconsistency or a failure of the benchmark's stated scope; it is a mismatch between the qualified evidence in Section 5.4 and the unqualified summary claims in the abstract and Practitioner's Guide. The RQ3 algebra is exact, but its clean covariance form uses E[w * 1[a=pi_e] | x] = 1, a known-propensity property that real logs with estimated propensities do not satisfy. Since the paper's purpose is to give practitioners guidance for real logged data, this transfer is load-bearing. A single targeted experiment with estimated propensities and a different nuisance class would settle whether the counterproductivity sign survives; until then, the headline claim should be conditioned on known or near-exact propensities and on the learner regimes actually tested. The core overlap-alignment and DR-stability findings are independently supported, so the appropriate adjustment is conditional acceptance rather than rejection.","tokens_in":37878,"tokens_out":9841,"duration_ms":97032,"concrete_test":"Run the RQ3 protocol of Section 5.4 on the synthetic and IHDP exact-value datasets with propensities estimated out-of-fold (LightGBM on x, as in Section 5.3) and with a low-capacity in-sample nuisance (strongly regularized linear model) in addition to LightGBM. Record mean signed bias for plain DR vs frozen-policy cross-fit and the sign of Cov(w * 1[a=pi_e], mu_hat_in) under known and estimated propensities. If the cross-fit is not more optimistic in the estimated-propensity cells, the unconditional wording in the abstract and Section 6 item 3 should be conditioned to the known-propensity regimes or the claim should be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.4 derives V_in minus V_honest as (1/n) sum (1 - w * 1[a=pi_e])(mu_hat_in - mu_hat_honest), whose expectation is -Cov(w * 1[a=pi_e], mu_hat_in) only because known propensities give E[w * 1[a=pi_e] | x] = 1. With estimated propensities this unit-expectation fails and propensity-error terms enter, so the identity no longer forces the same sign. The empirical support is eight regimes, seven sharing the IHDP covariate matrix and all using LightGBM as the in-sample learner with exact propensities; Section 5.4 explicitly says the covariance sign and materiality are empirical, and Section 4.1 notes the shared-covariate limitation. Nevertheless, the abstract ('cross-fitting the nuisance alone leaves the reuse bias in place and makes it worse') and Practitioner's Guide item 3 ('Nuisance-only cross-fitting worsened it') state the counterproductivity without the known-propensity qualifier. This is the load-bearing transfer to practice: a real deployment log has estimated propensities and possibly a different learner, and no evidence in the paper establishes that the sign persists there.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks when off-policy evaluation (OPE) of deterministic equal-cost top-k allocation policies can be trusted, and benchmarks six estimators (DM, IPS, SNIPS, DR, Switch-DR, mIPS) on five datasets plus two six-DGP hardening suites, keeping exact-value references (synthetic, IHDP, ACIC-style) separate from noisy Horvitz-Thompson references on randomized trials (Hillstrom, Lenta, Jobs). Three headline claims are made. First, weak overlap is governed by logger-target action alignment rather than logging sharpness alone: Proposition 1 gives an exact sharpening limit showing a score-aligned logger's flat overlap is a finite-range plateau, and the empirical reversal under a cutoff-centred logger confirms it; effective sample size ranks overlap risk across logging environments, but not within a single fixed log, and its cut points do not transfer. Second, under policy-evaluation reuse (the optimizer's curse), cross-fitting only the outcome nuisance leaves the reuse bias in place and, across the eight known-effect regimes studied, makes it worse, while honest policy-level splitting cuts bias by 58-92% at the price of targeting the learning procedure's estimand. Third, propensity-estimation error is the largest degradation measured, can invert the ESS diagnostic (AUC 0.05 under a marginal propensity model), and leaves DR essentially unchanged. A non-simulated paired reference (Twins) replicates all three mechanisms but not the calibrated cut points.","tokens_in":38077,"tokens_out":24531,"duration_ms":201220,"significance":"If the claims hold, the paper supplies a decision procedure practitioners can apply before trusting an OPE estimate of a top-k rule: assess logger-target alignment and propensity-model credibility, use ESS only for across-environment ranking, and answer policy-evaluation reuse with policy-level honesty rather than nuisance cross-fitting. The negative results are as valuable as the positive ones: the within-log failure of ESS, the non-transferability of cut points, and the inversion of the screen under a poor propensity model are concrete scoping results. Methodological strengths are substantial and partly machine-checked: a released benchmark with documented make targets and per-row git commit hashes; numerical agreement of four estimators with Open Bandit Pipeline to floating-point precision; a derived-and-then-tested sharpening limit (Proposition 1); leave-one-dataset-out and out-of-fold nuisance reruns; measured coverage of the paper's own conditional intervals (DM 0.89 vs nominal 0.95, reported); an external reference read off recorded paired outcomes (Twins) with an explicit account of what fails to transfer; and Appendix A tabulating every claim against its evidence.","major_comments":[{"comment":"The headline statements of the RQ3 result (abstract: 'cross-fitting the nuisance alone leaves the reuse bias in place and makes it worse'; conclusion: 'cross-fitting the nuisance alone makes the optimism worse'; Practitioner's Guide item 3: 'Nuisance-only cross-fitting worsened it') are stated without the known-propensity qualifier that the paper's own derivation imposes. In Section 5.4 the identity is V_in - V_honest = (1/n) sum_i (1 - w_i * 1[a_i=pi_e(x_i)])(mu_in - mu_honest)(x_i, pi_e(x_i)), and the text correctly notes that its expectation equals -Cov(w * 1[a=pi_e], mu_in) only 'once cross-fitting makes mu_honest independent of the evaluation draw and the propensities are known, so E[w * 1[a=pi_e]|x] = 1 — as they are here by construction.' The body is properly scoped ('We do not establish when the covariance is material'; Appendix A: 'under the benchmark's known propensities'), but the practitioner-facing deliverables are not. This is load-bearing because (i) every RQ3 test, including the Twins replication, uses propensities known by construction since all logging is synthesized; (ii) with estimated propensities E[w_hat * 1[a=pi_e]|x] = pi_b(pi_e(x)|x)/pi_b_hat(pi_e(x)|x) != 1, so an additional first-order propensity-error term enters whose sign is untested; and (iii) Section 5.3 shows propensity error is the dominant degradation in realizable settings, so the unqualified 'worse' is transferred to exactly the regime the paper itself flags as hardest. The actionable core of item 3 — under reuse, use honest policy-level splitting; nuisance-only cross-fitting does not remove the reuse bias — survives the sign question, so the necessary fix is local: qualify 'makes it worse' in the abstract, conclusion, and guide as holding under the benchmark's known-propensity, studied-learner regimes, and ideally add an RQ3 rerun with out-of-fold estimated propensities using the machinery already built in Section 5.3/Table 9.","section":"§5.4; abstract ¶2; §6 item 3; conclusion"}],"minor_comments":[{"comment":"The negated indicator notation '⊮[a_i=pi_e(x_i)]' is nonstandard and can be misread as a disjointness or not-an-element symbol; please write 1 - 1{a_i = pi_e(x_i)} or an explicit complement notation.","section":"§5.4, Appendix A"},{"comment":"The column header 'nuis. x-fit % removed' is awkward for rows in which the reported change is negative; a header such as '% change in |bias|' with a note that negative values are degradations would prevent misreading.","section":"Table 4"},{"comment":"The line 'RQ2 — the effect of logging sharpness depends on logger–target action alignment' immediately above Figure 1 appears to be a duplicate floating title, since the caption below it states the same point; it should be removed or merged into the caption.","section":"Figure 1"},{"comment":"The phrase 'the failure gradient sharpens to 0.0%/1.7%/26.7%' is hard to parse because the verb 'sharpens' collides with the paper's logging-sharpness terminology; consider 'the failure gradient becomes 0.0%/1.7%/26.7%' or 'steepens to'.","section":"§1, Contribution 1"},{"comment":"The sentence 'Effective sample size ... ranks this risk across logging environments — guidance for designing logs and choosing estimator families, since it is weak at ranking candidate policies within a single fixed log' would be clearer if the within-log limitation were its own sentence rather than a 'since' clause, which momentarily reads as if the limitation were the reason for the guidance.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a serious empirical benchmark paper with exemplary transparency — Section 4.1's scope list, Section 5.4's explicit 'we do not establish when the covariance is material,' Appendix A's claims table, measured interval coverage, and the full make-target reproduction chain all argue for publishing after a focused revision. My one substantive concern is that the abstract, conclusion, and Practitioner's Guide overstate the RQ3 sign relative to the body's known-propensity scoping; qualification is necessary, and an estimated-propensity rerun of RQ3 would strengthen the paper materially. The five-cluster inference limitation is real but handled honestly and should not block. I see no novelty-disclosure or scope problem: the manuscript cites the overlapping prior work (Tanaka et al. 2026; Douglas et al. 2026; Li 2026) explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, unusually honest empirical benchmark, and the headline claims mostly hold at the strength the paper states. The genuinely new piece is the RQ2 result—overlap risk in top-k allocation is governed by logger-target action alignment rather than logging sharpness—backed by Proposition 1's sharpening limit and confirmed by the cutoff-centred rerun. The RQ3 finding that nuisance-only cross-fitting makes reuse bias worse is well supported across the eight known-effect regimes, and the exact covariance identity is a nice analytic contribution. The paper also deserves credit for pairing every claim with evidence at a stated strength, listing five limits up front in Section 4.1, never pooling exact-value and HT-reference results, and validating mechanisms on the Twins reference, which uses recorded paired outcomes rather than a modelled surface. The released make targets and the numerical agreement check against Open Bandit Pipeline are real reproducibility evidence, and the negative results (within-log ESS near zero, non-transferable model-adequacy screen) are reported rather than buried.\n\nSoft spots, in proportion. The stress-test note is correct: the abstract and Practitioner's Guide item 3 say 'nuisance-only cross-fitting makes it worse' without the known-propensity qualifier that Section 5.4 itself flags. The identity's expectation is -Cov(w * indicator, mu_in) only because E[w * indicator | x] = 1 under exact propensities; with estimated propensities and a different learner, the sign is untested. That is a real gap between the stated claim and the evidence, and it is the load-bearing transfer to practice. It does not sink the paper—the authors are explicit that covariance sign and materiality are empirical—but the guide should carry the qualifier. Also, the effective number of informative datasets is closer to four, seven of eight RQ3 regimes share the IHDP covariate matrix, and all exact-value surfaces are synthetic or semi-synthetic; the paper says all this, but the cross-dataset generality is thinner than the 'eight regimes' phrasing suggests. The citation pattern looks fine; the one self-citation is contextual and not load-bearing.\n\nWho this is for: anyone doing OPE for deterministic allocation policies, and anyone building benchmarks for weak-overlap diagnostics. It deserves a serious referee; the right outcome is likely accept after a revision that tightens the RQ3 wording in the abstract and guide to match the paper's own scope statement. I would cite it for the alignment result and the RQ3 identity.","headline":"A careful, self-scoped benchmark; the RQ2 alignment result and RQ3 algebra are solid, but the guide's RQ3 wording overreaches the known-propensity setting.","tokens_in":38639,"tokens_out":1878,"would_cite":true,"duration_ms":18231,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that trust in offline evaluation of deterministic top-k allocation is governed by logger-target action alignment and is rankable in advance from logged data, whenever the propensity model is credible.","keywords":["off-policy evaluation","top-k allocation","weak overlap","effective sample size","doubly robust estimation","optimizer's curse","propensity estimation","offline evaluation benchmark"],"falsifier":"Run the RQ3 comparison on a real deployment log with estimated, not known, propensities and compute the sample covariance between the weight-times-action-match indicator and the in-sample outcome nuisance on the evaluation fold; if that covariance is zero or negative, nuisance-only cross-fitting will not worsen the reuse bias, and the paper's central RQ3 guidance fails for that regime.","tokens_in":37614,"feed_emoji":"🎯","tokens_out":11685,"duration_ms":91723,"temperature":0.7,"pith_summary":"Off-policy evaluation is meant to tell an organization what a budget-constrained targeting rule would earn before deployment, but a deterministic top-k rule removes all averaging over actions, so the importance weights become either zero or the reciprocal of the logger's probability of the rule's chosen action. This paper establishes that the decisive risk factor is the alignment between the logger and the rule's actions, not the sharpness of the logging policy: sharpening a score-aligned logger barely moves overlap, while an action-level disagreement collapses effective sample size and multiplies weighting error. It then shows the risk is rankable ex ante from logged data across logging environments, with effective sample size as a valid between-environment screen that is weak within a single log and whose cut points do not transfer, and that a credible propensity model is a prerequisite because propensity-estimation error is the largest measured degradation and can invert the screen. For the optimizer's curse, when the rule is learned on the evaluation data, cross-fitting only the outcome nuisance makes the optimistic bias worse, while honest policy-level splitting removes the reuse by targeting the learning procedure's value rather than the full-sample policy's value. Across the benchmark's stresses, doubly robust estimation is the most stable estimator, making it the defensible default when at least one nuisance model is credible.","feed_headline":"Top-k offline evaluation fails when logger and rule disagree","feed_subtitle":"Benchmark shows overlap risk is rankable in advance, and nuisance-only cross-fitting won't fix the optimizer's curse.","key_machinery":"The load-bearing object is the deterministic-target importance weight $w_i = \\pi_e(a_i|x_i)/\\pi_b(a_i|x_i)$, which takes only the values $0$ when the logger's action differs from the rule's, or $1/\\pi_b(a_i|x_i)$ when it matches, so every weak-overlap mechanism runs through the logger's probability of the rule's chosen action. The paper's overlap diagnostic, the normalized Kish effective-sample-size fraction, factors exactly into an action-coverage term times a dispersion term on matched units, which is why it ranks risk across logging environments but not candidates within one log. The optimizer's-curse argument is carried by an exact finite-sample identity, $\\widehat{V}_{\\mathrm{in}} - \\widehat{V}_{\\mathrm{honest}} = \\frac{1}{n}\\sum_i (1 - w_i \\mathbf{1}[a_i=\\pi_e(x_i)])(\\widehat{\\mu}_{\\mathrm{in}} - \\widehat{\\mu}_{\\mathrm{honest}})(x_i,\\pi_e(x_i))$, whose expectation under known propensities is $-\\mathrm{Cov}(w\\mathbf{1}[a=\\pi_e], \\widehat{\\mu}_{\\mathrm{in}})$; whether that covariance is positive and material is the empirical content the eight known-effect regimes test. A third named mechanism is the sharpening limit of Proposition 1, which gives the population ESS fraction limits for score-aligned and action-aligned loggers, approximately $\\varepsilon/(1-c)$ versus $1-\\varepsilon$.","core_discovery":"On the paper's own terms, the central claim is that weak-overlap failure in offline evaluation of equal-cost top-k allocation is a property of logger–target action alignment, not of logging sharpness: for a deterministic target the weight is $\\pi_e(a|x)/\\pi_b(a|x)\\in\\{0, 1/\\pi_b(a|x)\\}$, so the quantity that governs support is $\\pi_b(a_{\\pi_e}(x)|x)$, the logger's probability of the exact action the rule selects. A sharpening-limit argument shows a score-aligned logger sits on a finite plateau whose population effective-sample-size fraction approaches $\\varepsilon/(1-c)$, while an action-aligned logger sharpens toward full support; the benchmark confirms the reversal by re-centring the logger at the budget cutoff. The paper also claims that the optimizer's curse is not cured by nuisance-only cross-fitting: an exact finite-sample identity expresses the difference between in-sample and honest DR evaluation as a covariance between the weighted action-match indicator and the in-sample outcome nuisance, and in all eight known-effect regimes that covariance is positive, so frozen-policy cross-fitting is more optimistic than plain DR. Honest policy-level splitting reduces the reuse bias by 58–92% across those regimes, at the price of changing the estimand to the learning procedure's value. Finally, replacing the exact propensity with an out-of-fold estimate is the largest degradation measured, and a poor propensity model can invert the ESS diagnostic from useful to misleading; doubly robust estimation is essentially immune to that stress.","pith_inferences":["Going beyond the paper: the covariance identity suggests a practical diagnostic: compute the sample covariance between the weight-action-match indicator and the in-sample outcome nuisance on the evaluation fold; a non-positive value marks a regime where nuisance-only cross-fitting is not the paper's danger.","Going beyond the paper: because the sharpening limit shows score-aligned loggers sit on a plateau, evaluation designers should treat logger–target action alignment rather than temperature as the primary overlap axis; the paper notes its own first design swept temperature on a score-aligned logger.","Going beyond the paper: the propensity-model inversion suggests a sanity check: if a flexible propensity model makes the ESS screen report the supposedly safest cells as the most dangerous, the model is likely near-constant and the screen is measuring coverage rather than dispersion.","Going beyond the paper: because mechanisms transfer but calibrated cut points do not, a deployment team could calibrate its own ESS threshold on a small randomized or labeled pilot before relying on the screen."],"forward_implications":["A practitioner can compute the ESS fraction and support deficiency from logged actions and propensities and use them to rank logging situations before deployment, treating a low ESS as a reason to seek a better-aligned log or a model-based estimator rather than as a pass/fail certificate.","The same diagnostic should not be used to choose among candidate policies within the single log a practitioner already holds; within one log its median correlation with realized error is near zero.","When the targeting rule was learned from the same data used to evaluate it, nuisance-only cross-fitting should be avoided; honest policy-level splitting is the way to remove reuse bias, accepting that the estimand becomes the learning procedure's value.","Doubly robust estimation is the safer choice when outcome-model adequacy or overlap is uncertain, because it remains stable where direct modeling degrades and where fully weight-based IPS fails.","Policy selection requires a single shared log for all candidates; scoring each candidate on its own aligned log compares policy–logger pairs, not policies, and IPS tends to over-select the easiest-to-evaluate candidate."],"supporting_citations":[{"why":"Defines doubly robust policy evaluation and the OPE problem frame the benchmark adopts.","marker":"Dudík et al., 2011"},{"why":"Defines the self-normalized IPS estimator (SNIPS) compared in RQ1.","marker":"Swaminathan & Joachims, 2015"},{"why":"Defines Switch-DR, the hybrid whose tuning collapses to DR under the bounded-weight floor.","marker":"Wang et al., 2017"},{"why":"Formalizes the optimizer's curse that RQ3 measures and seeks to remove.","marker":"Smith & Winkler, 2006"},{"why":"Supplies the cross-fitting procedure whose nuisance-only variant RQ3 finds counterproductive.","marker":"Chernozhukov et al., 2018"},{"why":"Supplies honest policy learning via sample splitting, the alternative RQ3 recommends.","marker":"Athey & Wager, 2021"},{"why":"Supplies the effective-sample-size diagnostic used as the overlap risk screen.","marker":"Austin, 2011"},{"why":"Supplies the IHDP semi-synthetic benchmark data with known potential-outcome means.","marker":"Hill, 2011"}],"fun_headline_variants":["Overlap failure tied to logger-rule misalignment, not sharpness","Cross-fitting nuisance won't cure top-k optimizer's curse","Propensity error biggest threat to offline top-k evaluation","Honest splitting beats cross-fitting for top-k evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the in-sample outcome model is elevated precisely on the observations the estimator weights most heavily, and that this pattern holds for real deployment logs with estimated propensities; if that pattern is absent, the guidance that nuisance-only cross-fitting makes the optimizer's curse worse does not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Overlap failure tied to logger-rule misalignment, not sharpness","Cross-fitting nuisance won't cure top-k optimizer's curse","Propensity error biggest threat to offline top-k evaluation","Honest splitting beats cross-fitting for top-k evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1501,"prompt_tokens":1204,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":820,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":820,"tokens_out":297,"duration_ms":2660,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:08:18.397789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the RQ3 comparison on a real deployment log with estimated, not known, propensities and compute the sample covariance between the weight-times-action-match indicator and the in-sample outcome nuisance on the evaluation fold; if that covariance is zero or negative, nuisance-only cross-fitting will not worsen the reuse bias, and the paper's central RQ3 guidance fails for that regime.","supporting_citations":[],"review_version":1}