REVIEW 6 cited by
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
read the original abstract
Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers' decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to "peek" at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-arm bandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.
Forward citations
Cited by 6 Pith papers
-
A General Framework for Optimal Group Sequential Testing via Mixed-Integer Linear Programming
The authors propose an S-MILP framework that optimizes group sequential testing boundaries to achieve faster rejection of the null hypothesis compared to traditional methods while controlling type I and type II errors.
-
Betting on Bets: Anytime-Valid Tests for Stochastic Dominance
Introduces anytime-valid e-processes for first- and higher-order stochastic dominance that achieve power one and remain valid under continuous monitoring.
-
Betting on Bets: Anytime-Valid Tests for Stochastic Dominance
We derive powerful e-processes for sequential stochastic-dominance testing, with power-one guarantees for first- and higher-order dominance.
-
A General Framework for Optimal Group Sequential Testing via Mixed-Integer Linear Programming
An S-MILP optimization finds group sequential test boundaries that dominate Lan-DeMets, Pocock, and O'Brien-Fleming methods by spending alpha more aggressively early while still controlling error rates.
-
Betting on Bets: Anytime-Valid Tests for Stochastic Dominance
Anytime-valid nonparametric e-process tests for stochastic dominance yield power-one sequential procedures competitive with classical fixed-sample SD tests.
-
Efficient Sequential Evaluation of Large Language Models
A confidence-sequence framework for sequentially estimating an LLM's average benchmark accuracy under adaptive question selection, with growth-oriented sampling rules that in practice often lose to uniform sampling.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.