REVIEW 6 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
read the original abstract
Off-policy evaluation (OPE) aims to estimate the performance of hypothetical policies using data generated by a different policy. Because of its huge potential impact in practice, there has been growing research interest in this field. There is, however, no real-world public dataset that enables the evaluation of OPE, making its experimental studies unrealistic and irreproducible. With the goal of enabling realistic and reproducible OPE research, we present Open Bandit Dataset, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset is unique in that it contains a set of multiple logged bandit datasets collected by running different policies on the same platform. This enables experimental comparisons of different OPE estimators for the first time. We also develop Python software called Open Bandit Pipeline to streamline and standardize the implementation of batch bandit algorithms and OPE. Our open data and software will contribute to fair and transparent OPE research and help the community identify fruitful research directions. We provide extensive benchmark experiments of existing OPE estimators using our dataset and software. The results open up essential challenges and new avenues for future OPE research.
Forward citations
Cited by 6 Pith papers
-
Estimating Causal Effects from Data Generated by Stochastic Algorithms
Logging the features and relative probability of one unexposed item alongside the exposed item identifies causal effects of content features from stochastic algorithms even with unobserved confounders.
-
Decision-Calibrated Conformal Uncertainty for Pacing Decisions in Streaming Advertising
Decision-calibrated conformal uncertainty for pacing uses the support function of the signed policy sensitivity set to achieve smaller uncertainty radii on public datasets.
-
Asymptotically Log-Optimal Bayes-Assisted Confidence Sequences for Bounded Means
A Bayesian predictive model adaptively constructs asymptotically log-optimal confidence sequences for bounded means using test martingales.
-
Asymptotically Log-Optimal Bayes-Assisted Confidence Sequences for Bounded Means
A Bayesian predictive model adaptively selects martingale factors to construct asymptotically log-optimal confidence sequences for bounded means while preserving anytime validity under misspecification.
-
Fed-CausalDiff: Decoupled Synchronization for Federated Do-Simulation and Policy Evaluation
Fed-CausalDiff proposes decoupled synchronization in a federated causal diffusion model to improve do-simulation and policy-value estimation across heterogeneous decentralized datasets.
-
Exploitation Over Exploration: Unmasking the Bias in Linear Bandit Recommender Offline Evaluation
Greedy linear models without exploration consistently achieve top-tier performance in over 90% of offline dataset evaluations for linear bandit recommenders, with hyperparameter tuning favoring minimal exploration and...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.