Pith. sign in

REVIEW 2 cited by

Democratizing online controlled experiments at Booking.com

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1710.08217 v1 pith:5VS5XC7H submitted 2017-10-23 cs.HC

Raphael Lopez Kaufman , Jegar Pitchforth , Lukas Vermeer This is my paper · ORCID
classification cs.HC
keywords experimentsbookingexperimentationonlineinfrastructurecontrolleddevelopmentlarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

There is an extensive literature about online controlled experiments, both on the statistical methods available to analyze experiment results as well as on the infrastructure built by several large scale Internet companies but also on the organizational challenges of embracing online experiments to inform product development. At Booking.com we have been conducting evidenced based product development using online experiments for more than ten years. Our methods and infrastructure were designed from their inception to reflect Booking.com culture, that is, with democratization and decentralization of experimentation and decision making in mind. In this paper we explain how building a central repository of successes and failures to allow for knowledge sharing, having a generic and extensible code library which enforces a loose coupling between experimentation and business logic, monitoring closely and transparently the quality and the reliability of the data gathering pipelines to build trust in the experimentation infrastructure, and putting in place safeguards to enable anyone to have end to end ownership of their experiments have allowed such a large organization as Booking.com to truly and successfully democratize experimentation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Moving towards informative and actionable social media research

    cs.SI 2025-05 unverdicted novelty 4.0 of 10

    Social media research yields inconclusive causal findings due to system complexity, and progress requires mechanistic explanations that integrate observational and experimental approaches while recognizing their share...

  2. Maturity Framework for Enhancing Machine Learning Quality

    cs.LG 2025-02 conditional novelty 4.0 of 10

    A quality score and five-level maturity framework for ML systems, open-sourced and rolled out at Booking.com to track and improve ML quality.

Pith tools