Logging Policy Design for Off-Policy Evaluation

arxiv: 2605.15108 · v2 · pith:XCPSYI2Inew · submitted 2026-05-14 · 📊 stat.ML · cs.AI· cs.IR· cs.LG· stat.ME

Logging Policy Design for Off-Policy Evaluation

Connor Douglas , Joel Persson , Foster Provost This is my paper

Pith reviewed 2026-05-15 03:04 UTC · model grok-4.3

classification 📊 stat.ML cs.AIcs.IRcs.LGstat.ME

keywords off-policy evaluationlogging policyreward-coverage tradeoffOPE errorrecommendation systemsinformational regimestreatment selectionpolicy value estimation

0 comments p. Extension

pith:XCPSYI2I Add to your LaTeX paper

What is a Pith Number?

\usepackage{pith}
\pithnumber{XCPSYI2I}

Prints a linked pith:XCPSYI2I badge after your title and writes the identifier into PDF metadata. Compiles on arXiv with no extra files. Learn more

The pith

A unifying framework derives optimal logging policies that minimize off-policy evaluation error by balancing reward concentration against action coverage across known, unknown, and partial information regimes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to design logging policies that collect data allowing low-error estimates of a target policy's value, such as a recommender system, without deploying it live. It identifies a core reward-coverage tradeoff: putting more logging probability on high-reward actions lowers variance but risks leaving gaps for actions the target policy might choose. The work solves for the best logging policy in three standard cases—when the target and rewards are fully known at collection time, when they are unknown, and when only priors or noisy estimates are available—and supplies practical rules for firms that must choose among candidate systems.

Core claim

We characterize a fundamental reward-coverage tradeoff and propose a unifying framework for logging policy design and derive optimal policies in canonical informational regimes where the target policy and reward distribution are (i) known, (ii) unknown, and (iii) partially known through priors or noisy estimates at logging time.

What carries the argument

The reward-coverage tradeoff, which determines how logging probability mass should be allocated to minimize the combined variance and bias of standard OPE estimators.

If this is right

When the target policy and rewards are known, the optimal logging policy concentrates mass on high-reward actions the target is likely to select.
When both are unknown, the optimal policy spreads probability to guarantee coverage of every action the target might take.
When only priors or noisy estimates exist, the optimal policy interpolates between the known and unknown cases using the available information.
Firms evaluating multiple candidate recommenders can use the derived policies to collect data that yields more accurate offline comparisons.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same tradeoff logic could be used to design adaptive logging policies that update as rewards are observed during data collection.
The framework connects directly to problems of experimental design in causal inference when the goal is policy-value estimation rather than simple average treatment effects.
Simulation studies in recommendation environments could quantify how much OPE error drops when the derived policies replace standard uniform or epsilon-greedy logging.

Load-bearing premise

The three informational regimes accurately describe the knowledge available when the logging policy is chosen and the variance and bias formulas used for OPE estimators match real behavior.

What would settle it

A controlled simulation or field test in which the theoretically optimal logging policy produces higher mean squared error for the target policy value than a uniform random logging policy when the target policy and reward distribution are known in advance.

Figures

Figures reproduced from arXiv: 2605.15108 by Connor Douglas, Foster Provost, Joel Persson.

**Figure 2.** Figure 2: Informational settings for logging policy design . The two dimensions of information [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

**Figure 3.** Figure 3: Illustration of error across logging policy choices in [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

**Figure 4.** Figure 4: MSE as a function of the level of noise in the reward estimates ˆµ [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗

**Figure 5.** Figure 5: Effect of posterior shrinkage and reward prediction noise on MSE and policy value . A [PITH_FULL_IMAGE:figures/full_fig_p025_5.png] view at source ↗

**Figure 6.** Figure 6: MSE of IPW estimator for soft-greedy logging policy classes [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗

**Figure 7.** Figure 7: MSE of IPW estimator for soft-greedy logging policy classes [PITH_FULL_IMAGE:figures/full_fig_p054_7.png] view at source ↗

read the original abstract

Off-policy evaluation (OPE) estimates the value of a target treatment policy (e.g., a recommender system) using data collected by a different logging policy. It enables high-stakes experimentation without live deployment, yet in practice accuracy depends heavily on the logging policy used to collect data for computing the estimate. We study how to design logging policies that minimize OPE error for given target policies. We characterize a fundamental reward-coverage tradeoff: concentrating probability mass on high-reward actions reduces variance but risks missing signal on actions the target policy may take. We propose a unifying framework for logging policy design and derive optimal policies in canonical informational regimes where the target policy and reward distribution are (i) known, (ii) unknown, and (iii) partially known through priors or noisy estimates at logging time. Our results provide actionable guidance for firms choosing among multiple candidate recommendation systems. We demonstrate the importance of treatment selection when gathering data for OPE, and describe theoretically optimal approaches when this is a firm's primary objective. We also distill practical design principles for selecting logging policies when operational constraints prevent implementing the theoretical optimum.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper derives explicit optimal logging policies for OPE under known, unknown, and partial-knowledge regimes, which is a clean extension of standard variance formulas but may not survive estimator misspecification.

read the letter

The core contribution is a framework that optimizes the logging policy to minimize OPE error by balancing reward concentration against coverage of the target policy's actions. It supplies closed-form or derivable optima in the three regimes: fully known target and rewards, completely unknown, and partial information via priors or noisy estimates. This is new; most OPE papers fix the logging policy and focus on better estimators instead of designing the data collection step itself. The reward-coverage tradeoff is stated plainly and the practical principles at the end give firms something they can actually use when choosing among candidate policies for offline testing. The math appears to rest on standard IPS and DR variance expressions plugged into the objective, which keeps the derivations tractable. The soft spot is exactly the one the stress-test flags: if the deployed estimator uses clipping, a misspecified reward model, or other common tweaks, the policy that is optimal for the modeled objective need not minimize true error. The abstract gives no simulation or robustness checks, so it is unclear how large this gap is in practice. The paper is aimed at people who control logging in recommendation or bandit settings and want to improve downstream OPE accuracy. A reader who already knows the basic OPE variance formulas will follow the derivations without trouble. It is worth sending to peer review; the central claim is concrete and the framework is internally consistent even if the robustness question needs more attention in revision.

Referee Report

2 major / 2 minor

Summary. The paper proposes a unifying framework for designing logging policies that minimize off-policy evaluation (OPE) error for a given target policy. It characterizes a fundamental reward-coverage tradeoff and derives optimal logging policies under three informational regimes: (i) target policy and reward distribution known, (ii) both unknown, and (iii) partially known via priors or noisy estimates at logging time. The results are illustrated with practical guidance for firms selecting among candidate recommendation systems.

Significance. If the derivations hold under standard OPE estimators, the work provides theoretically grounded and actionable principles for data collection in OPE, addressing a practical gap in high-stakes settings such as recommender systems. The unification across knowledge regimes and explicit treatment of the reward-coverage tradeoff represent a clear contribution to the OPE literature.

major comments (2)

[§4] §4 (unknown regime): The minimax optimality derivation plugs the standard IPS/DR variance formula directly into the objective and solves over reward distributions. This yields a closed-form logging policy only under the exact variance expression; the paper does not show that the same policy remains optimal when the deployed estimator uses a misspecified reward model or clipped importance weights, which is the typical case in practice.
[§5] §5 (partial-knowledge regime): The optimality result relies on the prior or noisy estimate entering the objective exactly as modeled. No sensitivity analysis is provided for how errors in the prior propagate to the derived logging policy or to the resulting OPE error bound.

minor comments (2)

[Abstract] The abstract states that derivations exist but the main text should include at least one explicit equation (e.g., the objective in Eq. (3) or the closed-form policy in the known regime) to allow readers to verify the claimed optimality without reconstructing the algebra.
[§2] Notation for the logging policy π_log and target policy π_tgt should be introduced once in §2 and used consistently; several later sections re-define the same symbols.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The comments highlight important practical considerations for the derived logging policies. We address each major comment below and outline targeted revisions that clarify assumptions and strengthen applicability without altering the core theoretical contributions.

read point-by-point responses

Referee: [§4] §4 (unknown regime): The minimax optimality derivation plugs the standard IPS/DR variance formula directly into the objective and solves over reward distributions. This yields a closed-form logging policy only under the exact variance expression; the paper does not show that the same policy remains optimal when the deployed estimator uses a misspecified reward model or clipped importance weights, which is the typical case in practice.

Authors: We agree that the closed-form result in §4 is derived under the exact IPS/DR variance expression without misspecification or clipping. The manuscript positions this as the ideal theoretical benchmark for the reward-coverage tradeoff, consistent with standard OPE analysis. We will revise §4 to explicitly state these assumptions, add a paragraph discussing how the policy may serve as a robust initialization in practice, and include a brief remark that extensions to misspecified or clipped estimators are left for future work. This does not change the main result but improves clarity on scope. revision: yes
Referee: [§5] §5 (partial-knowledge regime): The optimality result relies on the prior or noisy estimate entering the objective exactly as modeled. No sensitivity analysis is provided for how errors in the prior propagate to the derived logging policy or to the resulting OPE error bound.

Authors: The partial-knowledge results treat the prior or noisy estimate as entering the objective in the modeled form, which enables the closed-form characterization. We acknowledge the value of sensitivity analysis for robustness. We will add a short subsection in §5 with a numerical sensitivity study (perturbing the prior mean/variance and reporting changes in the resulting logging policy and OPE bound) to quantify propagation of errors. This revision directly addresses the concern while remaining within the paper's scope. revision: yes

Circularity Check

0 steps flagged

No circularity: derivations optimize standard OPE variance expressions without self-referential reduction

full rationale

The paper's core derivations minimize an OPE error objective constructed from established IPS/DR variance and bias formulas applied to the logging policy probabilities, target policy, and reward distribution. These steps constitute a standard optimization problem over known mathematical expressions rather than redefining the target quantity in terms of itself or fitting parameters that are then relabeled as predictions. No self-citation chains, uniqueness theorems, or ansatzes from prior author work are invoked as load-bearing justifications; the informational regimes are treated as modeling assumptions under which the optimization is solved. The resulting policies are therefore independent outputs of the framework, not equivalent to the inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on standard domain assumptions about reward distributions and policy knowledge levels; no new free parameters or invented entities are described in the abstract.

axioms (1)

domain assumption Reward distributions and target policies exist and can be characterized as known, unknown, or partially known at logging time
Invoked to define the three canonical regimes in which optimal policies are derived.

pith-pipeline@v0.9.0 · 5493 in / 1170 out tokens · 55165 ms · 2026-05-15T03:04:05.796307+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We characterize a fundamental reward-coverage tradeoff... derive optimal policies in canonical informational regimes
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Neyman allocation... posterior shrinkage under Gaussian hierarchical prior

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.