Pith. sign in

REVIEW 3 major objections 7 minor 23 references

$t$-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A cheap resampling check can flag outcome metrics with unreliable t-test p-values.

desk verdict Honest, well-scoped screening idea for A/B testing assumptions, but the effectiveness claim is unsupported because the KS flag is never calibrated to empirical false-positive rates. read the letter →

arxiv 2502.04793 v2 pith:6SK2GLKH submitted 2025-02-07 stat.ME cs.LG

Olivier Jeunen This is my paper · ORCID
classification stat.MEcs.LG MSC 62F0362G10
keywords A/Btestingt-testassumptionsnormalityvalidationKolmogorov-Smirnovtestp-valueuniformitycentrallimittheoremTypeIerrorA/Atests
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the silent premise behind most A/B tests—that the Central Limit Theorem has made the sampling distribution of the average treatment effect normal—can be checked cheaply before any treatment is run. The proposed method repeatedly resamples A/A tests from logged user data, records the p-values that a normal-theory test produces under a known zero effect, and uses the Kolmogorov-Smirnov test to check whether those p-values look uniform. On a real dataset of about two million users and fifty user-event metrics, the procedure flags several metrics where normal confidence intervals would misstate uncertainty despite roughly one million users per group. A practitioner who takes the claim seriously would screen metrics with this diagnostic and switch to non-parametric intervals for the flagged ones.

What carries the argument

The machinery is the resampled A/A test combined with the Kolmogorov-Smirnov uniformity test. An A/A test is a random split of the same user population into two groups with no treatment, so the true average treatment effect is zero; under the null and the normality assumption the resulting z-scores are standard normal and the p-values are uniform. Repeating the split n times gives an empirical CDF of p-values, and the KS D-statistic measures the largest gap between that CDF and the uniform CDF, with the Kolmogorov distribution providing a p-value for the departure. When the KS p-value is low, the paper recommends treating normal-theory confidence intervals for that metric as unreliable.

What would settle it

Simulate user-level outcomes with a known skewed or zero-inflated distribution and a true effect of zero, run the proposed 5,000-resample A/A procedure, and compare KS flags against how often the stated 95% confidence intervals actually contain the true zero effect; if the KS test fails to flag metrics whose coverage is far from 95%, or flags metrics whose coverage is correct, the method's diagnostic value is refuted.

Watch

Extended reading notes

Core claim

The central claim is that the validity of the t-test's normal approximation is directly observable from data: if the sampling distribution of the average treatment effect has converged to normality, then p-values computed from zero-effect A/A comparisons are uniform, and any detectable departure from uniformity in those p-values is evidence that normal confidence intervals and p-values for that metric are not reliable. The paper demonstrates this by constructing 5,000 synthetic A/A comparisons for each of 50 event-type metrics, computing the Kolmogorov-Smirnov D-statistic between the empirical p-value distribution and the uniform distribution, and showing that the test catches metrics that would otherwise be trusted on sample-size grounds. Event frequency and skewness correlate with the D-statistic but do not determine it, so the KS check supplies information those summary statistics miss.

Load-bearing premise

The procedure treats the p-values from n repeated random splits of one fixed user population as independent and identically distributed, even though A/A groups built from the same users overlap and the resamples are therefore correlated.

Editorial extensions

If this is right

  • If the method is correct, a team with historical logs can pre-screen candidate metrics and exclude or re-estimate those whose p-value distributions are non-uniform before committing to an experiment.
  • The non-monotonic relationship with event frequency implies that large sample sizes alone do not certify t-test validity; practitioners need the diagnostic even when groups are huge.
  • The D-statistic adds independent information over skewness, so a metric can be low-frequency or skewed yet still pass, while a frequent and nearly symmetric metric can still fail.
  • Flagged metrics should be handled with permutation or bootstrap confidence intervals rather than normal intervals, at the cost of extra computation.
  • The uniformity check applies to any normal-theory estimator whose validity rests on asymptotic normality, not only the difference-in-means t-test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a natural extension is to run this screen on synthetic or historical baseline data before launch, treating the KS D-statistic as a metric-quality score and selecting the most trustworthy outcome metrics.
  • Editorial inference: the same uniformity logic could be applied to ratio metrics or delta-method estimators, where normality is even less guaranteed.
  • Editorial inference: the D-statistic could be monitored as more data accumulates, giving an empirical, metric-specific sample-size criterion for when the central limit theorem has 'kicked in'.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The manuscript proposes a practical diagnostic for deciding when normal-theory confidence intervals and p-values for average treatment effects are trustworthy in online A/B tests. The method repeatedly resamples A/A comparisons from a fixed user population, computes a z/t-test p-value for each resample, and applies a Kolmogorov-Smirnov test to check whether the resulting p-value distribution is uniform. The paper argues that uniformity is the property that guarantees correct CI coverage and false-positive rates, and it demonstrates the approach on a proprietary dataset with roughly two million users and fifty event types, reporting KS statistics and correlations with event frequency and skewness. Three research questions are posed and answered affirmatively based on the real-data example.

Significance. The core logical step—that p-values from valid A/A tests should be uniform and that uniformity can be checked with a KS test—is statistically sound, requires no fitted parameters, and is simple enough for practitioners. The paper is candid about important caveats such as multiple testing, the need for nonparametric alternatives, and the dangers of binary significance labels. If the diagnostic is properly calibrated to false-positive rates, it would be a useful addition to the A/B-testing toolkit. At present, however, the empirical evidence does not establish the paper's central claim that the KS test 'uncovers outcomes for which normal CIs are not appropriate,' because the connection to actual CI coverage and false-positive rates is never quantified.

major comments (3)
  1. [Section 3, RQ1, Figure 1] The central claim that the KS test on resampled A/A p-values 'uncovers outcomes Y for which normal CIs are not appropriate' is never validated against the quantity the method is meant to diagnose. The same 5,000 A/A resamples directly yield an empirical false-positive rate at any alpha (the proportion of p-values below alpha) and an empirical CI coverage rate; these should be reported for flagged and non-flagged metrics. With n=5,000, a D-statistic around 0.02 is already significant at the 5% level and may correspond to a false-positive rate at alpha=0.05 of 0.051, a practically irrelevant violation, while the sup-norm statistic can miss a concentrated lower-tail inflation unless it is the maximum deviation. As presented, a significant KS p-value is not evidence that the t-test is unreliable for practical decision-making, so the paper's RQ1 answer and the abstract's 'efficient and effective' claim are not supported.
  2. [Section 2.2] The validity of the KS test depends on the n p-values being an independent and identically distributed sample from the null distribution, but the manuscript does not specify the resampling protocol in enough detail to verify this. The text says only that 'we repeatedly resample groups A_i, A_i' for n iterations.' If each iteration is an independent random partition of the fixed population, the p-values are independent even though users appear in multiple resamples; if a different scheme is used, for example resampling from a small set of precomputed partitions, dependence could distort the KS null distribution. The authors should state the exact procedure and justify the i.i.d. assumption or use a method that accounts for dependence.
  3. [Section 3, RQ1] Because the real-data ground truth is unknown, the empirical demonstration cannot by itself show that the KS test correctly identifies metrics with inflated Type-I errors. A small simulation study with outcome distributions of known shape, such as heavy-tailed or highly skewed distributions, would allow the authors to compare KS flags against the true false-positive rates and CI coverage, and to characterize the operating characteristics of the D-statistic threshold. Such a study would directly substantiate the claim that the procedure 'identifies scenarios prone to inflated Type-I errors.'
minor comments (7)
  1. [Section 2.2] Please clarify whether A_i and A_i' are disjoint within each iteration and whether groups are sampled with or without replacement; the statement that each group has 'roughly 1 million users' appears only in the RQ2 discussion.
  2. [Section 3] There is a typo in the second paragraph: 'subet' should be 'subset.'
  3. [Figure 3] The text reports a Spearman correlation of approximately 0.43, but the figure shows only four events; the number of events used for the correlation should be stated.
  4. [Abstract and Section 2.1] The paper refers to the 't-test' throughout but uses standard normal quantiles in Equations (5) and (6); since the sample sizes are very large the distinction is immaterial, but the terminology should be aligned.
  5. [Figure 1] The footnote on multiple testing is useful; consider drawing a Bonferroni-corrected threshold directly on the figure to aid interpretation.
  6. [References] The Kolmogorov-Smirnov test is cited to a practitioner-oriented book; a standard statistics textbook reference for the KS null distribution would be more appropriate.
  7. [Equation (6)] The notation for the p-value formula is visually confusing; a standard expression such as p = 2(1 - Phi(|z|)) would be clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

The paper's KS-based uniformity diagnostic is derived from standard probability theory and is not circular.

full rationale

The derivation chain is self-contained. The paper starts from the standard fact that, under the null hypothesis, a valid test statistic yields p-values that are uniform: Eq. (6) defines p as a function of the z-score, and the uniform property follows from the probability integral transform. The KS test (Eq. 7) then provides an independent, externally grounded null distribution for the empirical CDF of those p-values. No parameter is fitted to the data being diagnosed, and no result is imported from prior work by the author to force the conclusion. The self-citations ([1], [12]--[14]) are contextual prior-work references and are not load-bearing. The remaining concerns, such as overlapping resamples potentially breaking the i.i.d. assumption for the KS test and the D-statistic not being calibrated to the empirical false-positive rate, are validity and effectiveness issues, not circular reductions. The central claim therefore does not reduce to its inputs by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The method introduces no fitted parameters and no new entities; it relies on standard CLT and KS theory plus an unstated domain assumption about independence of resampled p-values. The choice of n=5000 A/A iterations is an experimental design decision, not a fitted constant.

assumptions (3)
  • standard math The Central Limit Theorem guarantees that the sampling distribution of the average treatment effect converges to normality, justifying the z-interval in Equation 5.
    Invoked in Section 2.1 to justify normal confidence intervals and p-values; this is the background premise the paper aims to validate empirically.
  • domain assumption The n p-values obtained from repeated resampled A/A tests form an independent and identically distributed sample from the null distribution of the test statistic, so that the Kolmogorov-Smirnov test's null distribution applies.
    Required for the KS test in Section 2.2 to yield valid p-values. The paper does not address the dependence induced by repeatedly splitting the same fixed user population without replacement.
  • standard math Under the null hypothesis, the Kolmogorov-Smirnov statistic D follows the Kolmogorov distribution, as stated in Equation 7.
    Standard result from KS test theory, used to compute the p-values that flag non-uniformity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $t$-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing." pith.science (2026). https://pith.science/paper/6SK2GLKH

@misc{pith2026250204793,
  author       = {Pith},
  title        = {Pith review of: $t$-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SK2GLKH}},
  note         = {Machine review of arXiv:2502.04793}
}
abstract

A/B-tests are a cornerstone of experimental design on the web, with wide-ranging applications and use-cases. The statistical $t$-test comparing differences in means is the most commonly used method for assessing treatment effects, often justified through the Central Limit Theorem (CLT). The CLT ascertains that, as the sample size grows, the sampling distribution of the Average Treatment Effect converges to normality, making the $t$-test valid for sufficiently large sample sizes. When outcome measures are skewed or non-normal, quantifying what "sufficiently large" entails is not straightforward. To ensure that confidence intervals maintain proper coverage and that $p$-values accurately reflect the false positive rate, it is critical to validate this normality assumption. We propose a practical method to test this, by analysing repeatedly resampled A/A-tests. When the normality assumption holds, the resulting $p$-value distribution should be uniform, and this property can be tested using the Kolmogorov-Smirnov test. This provides an efficient and effective way to empirically assess whether the $t$-test's assumptions are met, and the A/B-test is valid. We demonstrate our methodology and highlight how it helps to identify scenarios prone to inflated Type-I errors. Our approach provides a practical framework to ensure and improve the reliability and robustness of A/B-testing practices.

Figures

Figures reproduced from arXiv: 2502.04793 by the authors.

Figure 2
Figure 2. Visualising the number of event observations over [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The empirical density function for various events [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 11 canonical work pages

  1. [1]

    Shubham Baweja, Neeti Pokharna, Aleksei Ustimenko, and Olivier Jeunen. 2024. Variance Reduction in Ratio Metrics for Efficient Online Experiments. In Proc. of the 46th European Conference on Information Retrieval (ECIR ’24) . Springer

    work page 2024

  2. [2]

    Alex Deng. 2015. Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments. In Proc.of the 24th International Conference on World Wide Web (WWW ’15 Companion) . ACM, 923–928. doi:10.1145/2740908.2742563

    work page arXiv 2015

  3. [3]

    Stevens, Tatiana Xifara, and Amit Gandhi

    Alex Deng, Luke Hagar, Nathaniel T. Stevens, Tatiana Xifara, and Amit Gandhi

    work page

  4. [4]

    Pavel Dmitriev, Somit Gupta, Dong Woo Kim, and Garnet Vaz. 2017. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Con- trolled Experiments. In Proc.of the 23rd ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining (KDD ’17) . ACM, 1427–1436. doi:10.1145/3097983.3098024

    arXiv 2017

  5. [5]

    Rick Durrett. 2019. Probability: Theory and Examples . Vol. 5th Ed. Cambridge university press

    work page 2019

  6. [6]

    Michael P Fay and Michael A Proschan. 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Surv 4 (2010), 1–39

    work page 2010

  7. [7]

    Hans Fischer. 2010. A History of the Central Limit Theorem: From Classical to Modern Probability Theory (1 ed.). Springer. 402 pages. doi:10.1007/978-0-387- 87857-7

    doi:10.1007/978-0-387- 2010

  8. [8]

    Ronald Aylmer Fisher. 1921. Statistical methods for research workers. Statistical methods for research workers. 1st Ed (1921)

    work page 1921

Show all 23 references
  1. [9]

    Senn, Kenneth J

    Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman, and Douglas G. Altman. 2016. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology 31, 4 (01 Apr 201...

    2016 doi

  2. [10]

    Gronau, Akash Raj K

    Quentin F. Gronau, Akash Raj K. N., and Eric-Jan Wagenmakers. 2021. Informed Bayesian Inference for the A/B Test. Journal of Statistical Software 100, 17 (2021), 1–39. doi:10.18637/jss.v100.i17

    2021 doi

  3. [11]

    Yongyi Guo, Dominic Coey, Mikael Konutgan, Wenting Li, Chris Schoener, and Matt Goldman. 2021. Machine Learning for Variance Reduction in Online Exper- iments. In Advances in Neural Information Processing Systems , Vol. 34. Curran Associates, Inc., 8637–8648

    2021

  4. [12]

    Olivier Jeunen. 2023. A Common Misassumption in Online Experiments with Machine Learning Models. SIGIR Forum 57, 1, Article 13 (dec 2023), 9 pages. doi:10.1145/3636341.3636358

    2023

  5. [13]

    Olivier Jeunen, Shubham Baweja, Neeti Pokharna, and Aleksei Ustimenko. 2024. Powerful A/B-Testing Metrics and Where to Find Them. In Proc.of the 18th ACM Conference on Recommender Systems (RecSys ’24) . ACM, 816–818. doi:10.1145/ 3640457.3688036

    2024

  6. [15]

    Ron Kohavi and Nanyu Chen. 2024. False Positives in A/B Tests. In Proc.of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24). ACM, 5240–5250. doi:10.1145/3637528.3671631

    2024

  7. [16]

    Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. 2014. Seven rules of thumb for web site experimenters. In Proc.of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’14) . ACM, 1857–1866. doi:10.1145/2623330.2623341

    2014

  8. [17]

    Ron Kohavi, Alex Deng, and Lukas Vermeer. 2022. A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. In Proc. of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22). ACM, 3168–3177. https://doi.org/10.1145/353467...

    2022

  9. [18]

    Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experi- ments: A practical guide to A/B testing . Cambridge University Press

    2020

  10. [19]

    Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel T. Stevens. 2024. Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology. The American Statistician 78, 2 (2024), 135–149. doi:10.1080/00031305.2023.2257237

    2024 arXiv

  11. [20]

    McShane, David Gal, Andrew Gelman, Christian Robert, and Jen- nifer L

    Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jen- nifer L. Tackett. 2019. Abandon Statistical Significance. The American Statistician 73, sup1 (2019), 235–245. doi:10.1080/00031305.2018.1527253

    2019 arXiv

  12. [21]

    Donald B Rubin. 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology 66, 5 (1974), 688

    1974

  13. [22]

    Juliet Popper Shaffer. 1995. Multiple Hypothesis Testing. Annual Review of Psychology 46, 1 (1995), 561–584. doi:10.1146/annurev.ps.46.020195.003021

    1995

  14. [23]

    Nilesh Tripuraneni, Lee Richardson, Alexander D’Amour, Jacopo Soriano, and Steve Yadlowsky. 2024. Choosing a Proxy Metric from Past Experiments. In Proc.of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24) . ACM, 5803–5812. doi:10.1145/3637528.3671543

    2024

  15. [24]

    Wasserstein, Allen L

    Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar. 2019. Moving to a World Beyond “p<0.05”. The American Statistician 73, sup1 (2019), 1–19. doi:10.1080/00031305.2019.1583913

    2019 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.