{"id":"ced10f09-21ff-4873-81be-4eb64e6d782b","arxiv_id":"2501.05278","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Continuous off-policy estimators give directionally correct predictions of payment-policy performance in dynamic auctions, enabling counterfactual comparison and in-sample policy optimization from logged data.","lead":"This paper tests whether off-policy evaluation estimators can predict how untested payment policies would perform in a dynamic auction marketplace, using only logged data from prior A/B tests. The authors report that continuous estimators beat discretized ones and that counterfactual policy comparisons matched the direction of a real held-out test, though not its magnitude.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central Y-vs-Z result is produced by an undefined reward-adjustment function f, not demonstrably by the OPE estimators the paper claims to validate; this gap must be closed before the direction-match can be read as evidence for OPE.","rationale":"The reader's verdict is REJECT with moderate confidence, and I agree with that disposition, but for a more precise reason than the reader's stated weakest assumption. The paper's central positive result—that OPE estimated the same directional lifts as the real Test-3—can only be evaluated if the computation that produced Figure 6 is fully specified. It is not: the reward calculator function f in Figure 2 is undefined, and no appendix equation maps to it. The standard OPE estimators in the appendix reweight logged rewards by ratios of action densities; they do not adjust the control-side reward by comparing predicted actions of two policies. Unless f is just a placeholder for the OPE reweighting, the experiment in Section III.C.1 may not involve those estimators at all. The second issue is the proxy densities: random forest regressors output point predictions, not conditional densities, and the paper does not explain how the kernel bandwidth in Eq. 6 is combined with the forest's predicted payment. Without held-out validation of X', Y', Z' (e.g., comparing predicted payments to logged payments), the importance weights are unverified. The reader's weakest assumption is proxy fidelity; I rate the undefined f as even more load-bearing because it blocks reproduction before proxy validation can even be attempted. Both are testable: a reimplementation using only the stated estimators, with explicitly defined density estimates, either reproduces the directional match or shows the match depended on unspecified components. Since the paper currently provides neither code nor data, the gap cannot be closed by inspection. For these reasons the REJECT verdict is appropriate; my read does not change it.","tokens_in":8613,"tokens_out":5679,"duration_ms":56815,"concrete_test":"Ask the authors to specify f and release the logged data (or an anonymized surrogate), then independently reproduce the Counterfactual Test-2 result using only the equations in Appendix A: train X', Y', Z' on training folds, estimate densities via KDE, compute continuous OPE estimates for Y vs Z on Test-2 data, and compare to Test-3. If the reproduced directional lifts differ from Figure 6 or do not match Test-3 within reasonable confidence intervals, the reported match is an artifact of the unspecified reward calculator or of the proxy densities, not a demonstration of the paper's estimators.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.C.1's Figure 6 is the only direct evidence that OPE recovers the sign of a real A/B test. The method that produced Figure 6, however, is not described by any equation in the paper. Section II.B/Figure 2 says a 'reward calculator' adjusts the observed A-side outcome O_AX to a counterfactual outcome O_AY' using an undefined function f(P_AX, P_AY'), where P_AX and P_AY' are actions predicted by policy X and by proxy policy Y'. None of the estimators in Appendix A (IPW, SNIPW, DM, DR, SNDR, or the KDE estimator in Eq. 6) contains such an adjustment. If f is responsible for the directional lift match, the result does not validate the OPE estimators the paper claims to compare. Additionally, the proxy policies X', Y', Z' provide the densities needed for importance weights, but the paper never states how a random-forest regressor/classifier is converted into a density, and never validates predicted actions against held-out logged actions. The central claim therefore rests on an unverifiable, unconnected computation; this is a missing-support gap at the exact point where the paper's conclusion depends.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies off-policy evaluation (OPE) estimators to logged data from three real A/B tests in a dynamic auction marketplace, with policies X, Y, and Z. The authors learn random-forest proxy policies for the logged policies, compare discrete and continuous OPE variants by mean absolute percentage error, and use OPE to construct a counterfactual Y-vs-Z comparison whose directional lifts are checked against an actual Y-vs-Z test. They also propose an off-policy learning procedure, OptPaL, that optimizes an OPE-based profit objective and is then evaluated against Policy X in a simulated test. The central claims are that continuous estimators reduce MAPE by about 20% relative to discrete ones, that SNDR performs best among the continuous estimators, that OPE recovers the directional lifts of a real A/B test, and that a policy optimized on OPE estimates improves resources and returns at comparable cost.","tokens_in":8761,"tokens_out":3523,"duration_ms":37224,"significance":"If the empirical claims were supported, the paper would provide a valuable industrial case study showing that OPE can rank payment policies in a dynamic auction environment before committing to a live experiment. The availability of three real A/B tests, including a ground-truth Y-vs-Z test, is a genuine strength, and the comparison of discrete versus continuous OPE estimators on real logged data addresses a practically important question. The paper also names several concrete components of a future analytics platform, which is useful framing. However, the current evidence does not establish the central claims: the only direct validation of OPE against a real test is produced by an unspecified 'reward calculator' that is absent from the estimator definitions, the proxy policies are never validated, and the OptPaL result is circular because the learned policy is evaluated with the same OPE objective used to train it. These gaps are load-bearing, so the paper in its present form is not yet a reliable basis for the stated conclusions.","major_comments":[{"comment":"The central Y-vs-Z validation rests on a 'reward calculator' that adjusts observed outcomes to counterfactual outcomes using an undefined function f(P_AX, P_AY'), but no equation in the paper or in Appendix A defines this function, and none of the estimators in Eqs. (1)-(6) contains such an adjustment. Since Figure 6 is the only direct evidence that OPE recovers the sign of a real A/B test, the paper must specify f, explain how it relates to the OPE estimators, and show the result separately for each estimator; otherwise the directional-lift match cannot be attributed to the OPE methods being compared.","section":"Section II.C.1 and Figure 6"},{"comment":"The proxy policies X', Y', and Z' are random-forest regressors/classifiers, yet the paper never states how a random-forest model is converted into the densities needed for importance weights (π_e(a|x)/π_b(a|x) in Eqs. (1)-(5) and Q_i in Eq. (6)). Moreover, the proxies are never validated against the actual actions taken in the logged data, so any bias or miscalibration in a proxy propagates into every OPE estimate and into the Y-vs-Z ranking. The authors should describe the density conversion explicitly and report held-out action-prediction accuracy for each proxy.","section":"Section II.C.2"},{"comment":"The OptPaL policy is trained by minimizing f = -(Returns_OPE(p) - Cost_OPE(p)), and its superiority over Policy X in Figure 8 is then reported using the same OPE estimates. This is circular: the comparison shows that a policy fitted to the estimator attains a high value of that estimator, which is not evidence of a genuinely better policy. An independent evaluation, for example on a held-out portion of logged data or against the actual Test-3 outcomes, is required before the claim that 'counterfactual policy learning helps us discover new optimal policies' can be supported.","section":"Section III.D and Figure 8"},{"comment":"The MAPE comparisons in Figures 4 and 5 are reported without confidence intervals, and the text indicates that hyperparameters such as the kernel K, bandwidth h, discretization bin count, and the random-forest/MLP hyperparameters are tuned with Optuna, apparently on the same evaluation data. This in-sample tuning makes the reported 20% MAPE improvement and the 'SNDR works best' conclusion difficult to assess. The authors should provide error bars or bootstrapped intervals and use a nested or held-out tuning procedure.","section":"Section III.A, III.B, Figures 4 and 5"}],"minor_comments":[{"comment":"The abstract repeatedly refers to 'resource allocation strategies', but the controllable units are payment policies; aligning the terminology with the problem formulation in Section II.A would improve readability.","section":"Abstract and Introduction"},{"comment":"The flowchart introduces the function f and the notation O_AX, O_AY', P_AX, and P_AY' but the caption does not explain the symbols; please add a legend or a detailed caption.","section":"Figure 2"},{"comment":"In Eq. (6), the notation Q_i = P(t_i ∥ x_i) uses a double bar; this should be the conditional probability P(t_i | x_i), and the text should clarify whether τ(x_i) is a deterministic action or a distribution over actions in the continuous case.","section":"Appendix A, Eq. (6)"},{"comment":"The statement that 'SNDR works the best' is made without any statistical comparison or effect-size information; please qualify it or provide uncertainty estimates.","section":"Section III.B"},{"comment":"The figures report lifts without confidence intervals or sample sizes, even though the text notes that a T-test is used for the actual tests; adding the intervals would make the directional-lift claim more interpretable.","section":"Figure 6 and Figure 8"},{"comment":"The conclusion states that the study 'demonstrated the effectiveness of off-policy evaluation', which overstates what is shown given the gaps in Sections II.C.1 and III.D; please temper the claim to match the evidence.","section":"Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript resembles an extended abstract and lacks the reproducibility details expected of a full journal paper: no code, no data description, no proxy validation, and no definition of the central reward-adjustment function. The circular OptPaL evaluation and the undefined f in the key validation figure are not merely stylistic problems; they would require new experiments and a substantially rewritten evaluation to address. I therefore recommend rejection, while noting that the real A/B-test setting and the question of continuous versus discrete OPE are potentially worth revisiting if the authors provide a complete and validated pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this is a short industry case study on off-policy evaluation for payment policies in a dynamic auction. It does something genuinely useful: it applies standard OPE estimators (IPW, SNIPW, DM, DR, SNDR, KDE) to a real setting with three A/B tests, and it checks the counterfactual Y-vs-Z comparison against an actual third test. That is a nice validation setup, and the observation that continuous OPE reduces MAPE by about 20% over discretized is plausible. The OptPaL policy learning is a natural extension, though not novel.\n\nThe paper's main contribution is the demonstration that directional lifts can be recovered offline. But here I have to agree with the stress-test note: the key Figure 6 is produced by a reward calculator with an undefined function f(P_AX, P_AY'). None of the estimators in the appendix contain such an adjustment. So the reader cannot tell whether the direction-match is due to OPE or to some proprietary adjustment. This is a missing-support gap at the exact load-bearing point. Also, the proxy policies X', Y', Z' are random forests, but the paper never says how a RF output is converted into a density for importance weights, nor validates proxies against held-out actions. That is another hole.\n\nThe OptPaL result is circular: the policy is trained to maximize Returns_OPE - Cost_OPE and then reported as better using those same estimates, with no held-out test. The MAPE comparisons lack confidence intervals, and the kernel/bandwidth tuning is done on the evaluation data. These are real limitations.\n\nNow, proportional credit: the writing is clear, the problem is relevant, and the authors are appropriately cautious in saying 'directional lifts' rather than exact values. The paper does not oversell the method beyond feasibility. But the feasibility claim is not yet substantiated because the crucial computation is not specified.\n\nWho is this for? Practitioners in marketplace design who want to see OPE applied to payment policies, and OPE researchers looking for a real-world case study. It is not a methodological contribution.\n\nMy recommendation: send to peer review, but insist on major revision. The authors need to define f precisely, show how densities are derived from the proxies, validate the proxies, provide code and anonymized data, and either run a live test of the learned policy or drop the optimal-policy claim. With those changes, this could be a credible feasibility study. As it stands, the central evidence is not reproducible.","headline":"A useful industrial case study whose central counterfactual result is not reproducible from the paper as written because the key adjustment step is undefined.","tokens_in":9403,"tokens_out":2742,"would_cite":false,"duration_ms":25285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Given logged data from two A/B tests that share a control policy, the paper argues that off-policy estimators can compare two treatment policies directly and recover the same directional lifts later observed in a real test.","keywords":["off-policy evaluation","counterfactual inference","dynamic auctions","payment policies","A/B testing","continuous treatment effects","kernel density estimation","policy optimization"],"falsifier":"Run a new A/B test comparing $Y$ and $Z$ in a market segment not used for training the proxies and check the sign of each metric's lift against the OPE prediction; if the predicted and observed directions disagree on even one headline metric, the claim that OPE can rank policies from logged data fails in that setting.","tokens_in":8280,"feed_emoji":"📊","tokens_out":9164,"duration_ms":77934,"temperature":0.7,"pith_summary":"The paper asks whether a platform can choose between two auction payment policies, $Y$ and $Z$, using only logged data from two earlier A/B tests in which both were compared against a common control $X$, without running a third live test. It argues that off-policy evaluation (OPE) can answer this: replacing $X$ with a learned proxy of $Y$ in the $Y$-versus-$Z$ comparison produced directional lifts that matched the real $Y$-versus-$Z$ test the authors later ran. The practical payoff is that payment policies in dynamic auctions can be screened and ranked offline, so fewer expensive A/B tests are needed and poorly performing policies are less likely to be exposed to users. The same machinery, made differentiable through continuous kernel-density estimators, is then used to learn a new payment policy that improves returns for the same cost.","feed_headline":"Two old A/B tests predict the auction policy worth testing next","feed_subtitle":"Off-policy evaluation recovered the same directional lifts as a later live A/B test, so fewer tests are needed.","key_machinery":"The load-bearing mechanism is the continuous kernel-density off-policy estimator, which turns logged context-payment pairs into importance weights: $\\hat{v}_{\\tau}=\\frac{1}{nh}\\sum_{i}K\\!\\left(\\frac{\\tau(x_i)-t_i}{h}\\right)\\frac{y_i}{Q_i}$, with bandwidth $h$, kernel $K$, proxy policy action $\\tau(x_i)$, and logged density $Q_i$. Random-forest classifiers and regressors supply the proxy policies $X'$, $Y'$, $Z'$ used to compute those ratios, and the continuous version makes the estimator differentiable so an MLP named OptPaL can be trained by gradient descent on a profit-based loss. The same pipeline supports discrete estimators, whose coarser binning is the comparison baseline.","core_discovery":"On its own terms, the paper's central discovery is that continuous off-policy estimators recover the sign of treatment effects between two auction payment policies from past A/B-test logs. Using the data from Test-1 ($X$ versus $Y$) and Test-2 ($X$ versus $Z$), the authors build proxy policies $X'$, $Y'$, $Z'$ and a counterfactual Test-2 in which Proxy $Y$ replaces $X$; the resulting lifts for cost, reach, resources, and returns tracked the lifts later measured in a real Test-3 ($Y$ versus $Z$). They also report that continuous OPE reduces mean absolute percentage error by roughly 20 percent relative to discretized OPE, that the self-normalized doubly robust estimator performs best among the continuous evaluators, and that a differentiable continuous estimator can train an MLP policy (OptPaL) whose simulated allocation matches Policy $X$'s cost and reach while increasing resources and returns.","pith_inferences":["The paper leaves implicit a validation step the ranking depends on: checking each proxy policy against the actual logged actions before trusting any OPE estimate; without that check, proxy bias is invisible in the comparison.","The same compare-two-treatments-through-a-common-control trick should generalize to ranking many policies from pairwise historical tests, provided the proxy models stay calibrated across contexts.","The reported success is stated in terms of directional lift; if a decision-maker needs the magnitude of lift rather than just its sign, the estimators would need separate validation on level accuracy."],"forward_implications":["Payment policies can be ranked offline before committing to a live test, reducing the number of A/B tests needed.","Continuous OPE gives roughly 20 percent lower mean absolute percentage error than discretized OPE across the metrics considered.","The self-normalized doubly robust estimator with a tuned kernel and bandwidth is the most reliable of the continuous evaluators tested.","A counterfactually optimized policy (OptPaL) found by gradient descent can match an existing policy's cost and reach while increasing simulated resources and returns."],"supporting_citations":[{"why":"Supplies the OPE pipeline that the paper adapts to the auction setting.","marker":"[4]"},{"why":"Provides the continuous-treatment kernel density estimator that carries the main evaluation.","marker":"[13]"},{"why":"Defines the doubly robust estimator family from which the paper's main baselines and the SNDR variant are built.","marker":"[6]"},{"why":"Introduces inverse probability weighting used as the foundational baseline estimator.","marker":"[5]"},{"why":"Introduces self-normalized importance weighting used by SNIPW and SNDR.","marker":"[8]"},{"why":"Provides self-normalized and intrinsically efficient estimators used as additional baselines.","marker":"[9]"}],"fun_headline_variants":["Off-policy eval predicts auction test lifts from old A/B logs","Continuous OPE beats discrete in auction policy simulations","Self-normalized DR best for forecasting auction policy gains","Two past A/B tests anticipate the next auction policy winner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proxy policies learned with random forests are faithful substitutes for the true payment policies; if a proxy's action distribution is biased, every importance weight, OPE estimate, and the Y-versus-Z ranking inherits that bias.","fun_headline_variants_meta":{"raw":{"variants":["Off-policy eval predicts auction test lifts from old A/B logs","Continuous OPE beats discrete in auction policy simulations","Self-normalized DR best for forecasting auction policy gains","Two past A/B tests anticipate the next auction policy winner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1412,"prompt_tokens":945,"completion_tokens":467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":401}},"tokens_in":561,"tokens_out":467,"duration_ms":5652,"temperature":1.0,"reasoning_tokens":401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:58.226299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a new A/B test comparing $Y$ and $Z$ in a market segment not used for training the proxies and check the sign of each metric's lift against the OPE prediction; if the predicted and observed directions disagree on even one headline metric, the claim that OPE can rank policies from logged data fails in that setting.","supporting_citations":[{"cited_title":"Policy evaluation and optimiza- tion with continuous treatments","cited_arxiv_id":null,"evidence_quote":"Provides the continuous-treatment kernel density estimator that carries the main evaluation."},{"cited_title":"Doubly robust policy evaluation and optimization","cited_arxiv_id":null,"evidence_quote":"Defines the doubly robust estimator family from which the paper's main baselines and the SNDR variant are built."},{"cited_title":"Learning from logged implicit exploration data","cited_arxiv_id":null,"evidence_quote":"Introduces inverse probability weighting used as the foundational baseline estimator."},{"cited_title":"The self-normalized estimator for counterfactual learning","cited_arxiv_id":null,"evidence_quote":"Introduces self-normalized importance weighting used by SNIPW and SNDR."},{"cited_title":"Intrinsically efficient, stable, and bounded off-policy evaluation for reinforcement learn- ing","cited_arxiv_id":null,"evidence_quote":"Provides self-normalized and intrinsically efficient estimators used as additional baselines."}],"review_version":1}