{"id":"94c6cf9b-6dc3-442d-b4d5-056e0573eaba","arxiv_id":"2502.04793","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A/A-test resampling with a Kolmogorov-Smirnov test on the resulting p-values is proposed as an empirical check for whether normal-approximation confidence intervals and p-values are valid for a given outcome metric.","lead":"This paper proposes a practical check for whether a t-test on an outcome metric is trustworthy: repeatedly split users into two identical groups, collect p-values, and test if those p-values are uniformly distributed using the Kolmogorov-Smirnov test. It applies the check to real user-event data and shows that some metrics fail, meaning their type-I error rates may be inflated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KS significance is never calibrated to actual false-positive inflation, so the claimed 'effectiveness' of the diagnostic is unsupported.","rationale":"The theoretical logic of the paper is sound: under the null and the normal approximation, the p-values from independent A/A resamples are iid uniform, and the KS test then provides a valid test of that uniformity. The paper's central claim, however, is not just that KS tests uniformity, but that this provides an efficient and effective way to assess whether the t-test's assumptions are met. The empirical section demonstrates that KS flags some events, but it never demonstrates that those flags correspond to practically meaningful failures of the t-test, such as materially inflated false positive rates or degraded CI coverage. Because the KS test is a consistent test, with n=5000 it will reject for small deviations that may have no practical impact; and because D is a global sup-norm statistic, it may miss deviations concentrated in one tail, which is exactly where false-positive inflation at conventional alpha levels lives. This is an addressable but genuine gap in the evidence supporting the central claim. The reader's identified weakest assumption about dependence among overlapping splits is, under the natural interpretation of independent random partitions of a fixed population, not actually a problem: conditional on the fixed data, the resampled p-values are iid functions of independent partition indicators. Therefore the main load-bearing concern is the lack of calibration of the KS result to the directly estimable false positive rate, not the i.i.d. assumption. The recommended check would settle this by using the paper's own resampled p-values to compare KS decisions with empirical FPR classifications. Since the appropriate verdict remains conditional pending this validation, the reader's verdict is unchanged.","tokens_in":6716,"tokens_out":16017,"duration_ms":188745,"concrete_test":"Reproduce the paper's 5000 A/A resamples for all 50 event types. For each event, compute the empirical false positive rate at alpha=0.05, FPR = #{p < 0.05}/5000, with Monte Carlo standard error sqrt(0.05*0.95/5000) approx 0.0031, and the KS test decision at the 5% level with a Bonferroni correction. Then cross-tabulate KS rejections against a practical classification, e.g., flag an event as practically problematic if FPR is outside [0.04, 0.06]. Report sensitivity and specificity of KS rejection for this classification. Also examine events with lower-tail inflation, e.g., #{p < 0.01}/5000 > 0.015, to see whether KS detects them. If KS rejections align well with practically problematic FPRs and KS non-rejections have acceptable FPRs, the central claim is supported; otherwise the diagnostic needs calibration or a tail-specific test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's RQ1 claim—that the KS test on resampled A/A tests can uncover outcomes for which normal CIs are not appropriate—is never checked against the quantity it is meant to diagnose. The same 5000 A/A resamples directly yield an empirical false positive rate for any alpha (proportion of p-values below alpha) and an empirical CI coverage rate; these are the quantities that actually matter for A/B testing. A significant KS result only means the p-value eCDF differs from uniform by more than random fluctuation under iid sampling. With n=5000, a D-statistic around 0.02 is already significant at the 5% level, and such deviations may correspond to a false positive rate at alpha=0.05 of 0.051—a practically irrelevant violation. Conversely, because the KS D-statistic is a sup-norm over all p, a distribution with an inflated lower tail but otherwise matching uniform may go undetected unless the maximum deviation is large. Thus the KS p-value or D-statistic is not directly interpretable as 'normal CIs are inappropriate.' The note that the i.i.d. assumption for the KS test is satisfied if each A/A partition is drawn independently from the fixed population, so the reader's dependence concern is not the main issue; the missing calibration to empirical false positive rates is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a practical diagnostic for deciding when normal-theory confidence intervals and p-values for average treatment effects are trustworthy in online A/B tests. The method repeatedly resamples A/A comparisons from a fixed user population, computes a z/t-test p-value for each resample, and applies a Kolmogorov-Smirnov test to check whether the resulting p-value distribution is uniform. The paper argues that uniformity is the property that guarantees correct CI coverage and false-positive rates, and it demonstrates the approach on a proprietary dataset with roughly two million users and fifty event types, reporting KS statistics and correlations with event frequency and skewness. Three research questions are posed and answered affirmatively based on the real-data example.","tokens_in":6923,"tokens_out":9496,"duration_ms":111712,"significance":"The core logical step—that p-values from valid A/A tests should be uniform and that uniformity can be checked with a KS test—is statistically sound, requires no fitted parameters, and is simple enough for practitioners. The paper is candid about important caveats such as multiple testing, the need for nonparametric alternatives, and the dangers of binary significance labels. If the diagnostic is properly calibrated to false-positive rates, it would be a useful addition to the A/B-testing toolkit. At present, however, the empirical evidence does not establish the paper's central claim that the KS test 'uncovers outcomes for which normal CIs are not appropriate,' because the connection to actual CI coverage and false-positive rates is never quantified.","major_comments":[{"comment":"The central claim that the KS test on resampled A/A p-values 'uncovers outcomes Y for which normal CIs are not appropriate' is never validated against the quantity the method is meant to diagnose. The same 5,000 A/A resamples directly yield an empirical false-positive rate at any alpha (the proportion of p-values below alpha) and an empirical CI coverage rate; these should be reported for flagged and non-flagged metrics. With n=5,000, a D-statistic around 0.02 is already significant at the 5% level and may correspond to a false-positive rate at alpha=0.05 of 0.051, a practically irrelevant violation, while the sup-norm statistic can miss a concentrated lower-tail inflation unless it is the maximum deviation. As presented, a significant KS p-value is not evidence that the t-test is unreliable for practical decision-making, so the paper's RQ1 answer and the abstract's 'efficient and effective' claim are not supported.","section":"Section 3, RQ1, Figure 1"},{"comment":"The validity of the KS test depends on the n p-values being an independent and identically distributed sample from the null distribution, but the manuscript does not specify the resampling protocol in enough detail to verify this. The text says only that 'we repeatedly resample groups A_i, A_i' for n iterations.' If each iteration is an independent random partition of the fixed population, the p-values are independent even though users appear in multiple resamples; if a different scheme is used, for example resampling from a small set of precomputed partitions, dependence could distort the KS null distribution. The authors should state the exact procedure and justify the i.i.d. assumption or use a method that accounts for dependence.","section":"Section 2.2"},{"comment":"Because the real-data ground truth is unknown, the empirical demonstration cannot by itself show that the KS test correctly identifies metrics with inflated Type-I errors. A small simulation study with outcome distributions of known shape, such as heavy-tailed or highly skewed distributions, would allow the authors to compare KS flags against the true false-positive rates and CI coverage, and to characterize the operating characteristics of the D-statistic threshold. Such a study would directly substantiate the claim that the procedure 'identifies scenarios prone to inflated Type-I errors.'","section":"Section 3, RQ1"}],"minor_comments":[{"comment":"Please clarify whether A_i and A_i' are disjoint within each iteration and whether groups are sampled with or without replacement; the statement that each group has 'roughly 1 million users' appears only in the RQ2 discussion.","section":"Section 2.2"},{"comment":"There is a typo in the second paragraph: 'subet' should be 'subset.'","section":"Section 3"},{"comment":"The text reports a Spearman correlation of approximately 0.43, but the figure shows only four events; the number of events used for the correlation should be stated.","section":"Figure 3"},{"comment":"The paper refers to the 't-test' throughout but uses standard normal quantiles in Equations (5) and (6); since the sample sizes are very large the distinction is immaterial, but the terminology should be aligned.","section":"Abstract and Section 2.1"},{"comment":"The footnote on multiple testing is useful; consider drawing a Bonferroni-corrected threshold directly on the figure to aid interpretation.","section":"Figure 1"},{"comment":"The Kolmogorov-Smirnov test is cited to a practitioner-oriented book; a standard statistics textbook reference for the KS null distribution would be more appropriate.","section":"References"},{"comment":"The notation for the p-value formula is visually confusing; a standard expression such as p = 2(1 - Phi(|z|)) would be clearer.","section":"Equation (6)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable if the calibration gap is addressed. Because the dataset is proprietary, independent verification is limited; I encourage the editor to request that the authors include simulations and, if possible, release code. The citation practice appears appropriate and I have no concerns about novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clear, well-scoped proposal for a pre-test diagnostic: repeatedly split data into A/A groups, collect t-test p-values, and KS-test those p-values for uniformity. The specific packaged procedure is new as far as the cited literature goes, even though every ingredient is standard. The writing is direct and the real-data demonstration (50 event types, about 2 million users) is illustrative.\n\nWhat it does well: the three research questions are sensible and the empirical answers support the claim that the diagnostic carries independent information beyond event frequency or skewness. The footnote acknowledging that direct interpretation of KS p-values needs multiple-testing correction is a nice touch.\n\nThe soft spot is calibration. The paper claims the approach 'identifies scenarios prone to inflated Type-I errors' and 'uncovers outcomes for which normal CIs are not appropriate,' but it never checks a KS flag against the empirical false positive rate—even though the same 5,000 resamples directly provide that rate as the proportion of p-values below a given alpha. A significant KS result can be driven by deviations in the middle of the distribution, which are immaterial for typical alpha thresholds; conversely, a lower-tail inflation can be missed because the D-statistic is a sup norm over all p. Without a scatterplot of KS p-value versus empirical false positive rate at, say, alpha=0.05, the effectiveness claim is unsupported. This is fixable: add that direct comparison for flagged versus non-flagged metrics.\n\nA secondary issue is that the resamples are random splits of the same fixed population, so the p-values are not strictly independent across iterations. With 2 million users the dependence is likely weak, but it should be addressed. Missing code and data is a minor annoyance given the proprietary data.\n\nThe central statistical logic—under the null with an exact test, p-values are uniform, and KS tests that—is sound. The paper is an honest, useful contribution. It deserves peer review, but in its current form I would not call the method 'effective' without the calibration analysis. My recommendation: conditional acceptance with a required calibration section.","headline":"Honest, well-scoped screening idea for A/B testing assumptions, but the effectiveness claim is unsupported because the KS flag is never calibrated to empirical false-positive rates.","tokens_in":7440,"tokens_out":2840,"would_cite":true,"duration_ms":31335,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A cheap resampling check can flag outcome metrics with unreliable t-test p-values.","keywords":["A/B testing","t-test assumptions","normality validation","Kolmogorov-Smirnov test","p-value uniformity","central limit theorem","Type I error","A/A tests"],"falsifier":"Simulate user-level outcomes with a known skewed or zero-inflated distribution and a true effect of zero, run the proposed 5,000-resample A/A procedure, and compare KS flags against how often the stated 95% confidence intervals actually contain the true zero effect; if the KS test fails to flag metrics whose coverage is far from 95%, or flags metrics whose coverage is correct, the method's diagnostic value is refuted.","tokens_in":6500,"feed_emoji":"📊","tokens_out":7026,"duration_ms":75089,"temperature":0.7,"pith_summary":"This paper argues that the silent premise behind most A/B tests—that the Central Limit Theorem has made the sampling distribution of the average treatment effect normal—can be checked cheaply before any treatment is run. The proposed method repeatedly resamples A/A tests from logged user data, records the p-values that a normal-theory test produces under a known zero effect, and uses the Kolmogorov-Smirnov test to check whether those p-values look uniform. On a real dataset of about two million users and fifty user-event metrics, the procedure flags several metrics where normal confidence intervals would misstate uncertainty despite roughly one million users per group. A practitioner who takes the claim seriously would screen metrics with this diagnostic and switch to non-parametric intervals for the flagged ones.","feed_headline":"Resampled A/A tests plus a KS check flags unreliable p-values","feed_subtitle":"Checking that null p-values are uniform catches metrics where normal confidence intervals mislead.","key_machinery":"The machinery is the resampled A/A test combined with the Kolmogorov-Smirnov uniformity test. An A/A test is a random split of the same user population into two groups with no treatment, so the true average treatment effect is zero; under the null and the normality assumption the resulting z-scores are standard normal and the p-values are uniform. Repeating the split n times gives an empirical CDF of p-values, and the KS D-statistic measures the largest gap between that CDF and the uniform CDF, with the Kolmogorov distribution providing a p-value for the departure. When the KS p-value is low, the paper recommends treating normal-theory confidence intervals for that metric as unreliable.","core_discovery":"The central claim is that the validity of the t-test's normal approximation is directly observable from data: if the sampling distribution of the average treatment effect has converged to normality, then p-values computed from zero-effect A/A comparisons are uniform, and any detectable departure from uniformity in those p-values is evidence that normal confidence intervals and p-values for that metric are not reliable. The paper demonstrates this by constructing 5,000 synthetic A/A comparisons for each of 50 event-type metrics, computing the Kolmogorov-Smirnov D-statistic between the empirical p-value distribution and the uniform distribution, and showing that the test catches metrics that would otherwise be trusted on sample-size grounds. Event frequency and skewness correlate with the D-statistic but do not determine it, so the KS check supplies information those summary statistics miss.","pith_inferences":["Editorial inference: a natural extension is to run this screen on synthetic or historical baseline data before launch, treating the KS D-statistic as a metric-quality score and selecting the most trustworthy outcome metrics.","Editorial inference: the same uniformity logic could be applied to ratio metrics or delta-method estimators, where normality is even less guaranteed.","Editorial inference: the D-statistic could be monitored as more data accumulates, giving an empirical, metric-specific sample-size criterion for when the central limit theorem has 'kicked in'."],"forward_implications":["If the method is correct, a team with historical logs can pre-screen candidate metrics and exclude or re-estimate those whose p-value distributions are non-uniform before committing to an experiment.","The non-monotonic relationship with event frequency implies that large sample sizes alone do not certify t-test validity; practitioners need the diagnostic even when groups are huge.","The D-statistic adds independent information over skewness, so a metric can be low-frequency or skewed yet still pass, while a frequent and nearly symmetric metric can still fail.","Flagged metrics should be handled with permutation or bootstrap confidence intervals rather than normal intervals, at the cost of extra computation.","The uniformity check applies to any normal-theory estimator whose validity rests on asymptotic normality, not only the difference-in-means t-test."],"supporting_citations":[{"why":"Supplies the standard A/B-testing reference and the statement that p-values should be uniform under the null hypothesis (cited at p. 188).","marker":"[18]"},{"why":"Provides the central limit theorem foundation that justifies treating the sample mean's distribution as normal in large samples.","marker":"[5]"},{"why":"Motivates the need to detect inflated false-positive rates in A/B tests, which the proposed diagnostic targets.","marker":"[15]"},{"why":"Provides the skewness and sample-size heuristics that the paper compares against the D-statistic in RQ3.","marker":"[16]"},{"why":"Supplies the intuition that skewness is an important diagnostic for CLT appropriateness, used as a baseline in the empirical analysis.","marker":"[17]"},{"why":"Justifies why the Mann-Whitney U-test is not a drop-in replacement for the t-test, because it tests a different null hypothesis.","marker":"[6]"},{"why":"Supports the footnote about multiple-testing correction when interpreting Kolmogorov-Smirnov p-values across many metrics.","marker":"[22]"}],"fun_headline_variants":["Check p-value uniformity to verify A/B test validity","KS test on A/A resamples catches unreliable t-tests","A/A resampling plus KS test flags skewed p-values","Validate A/B tests by checking null p-values are uniform"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The procedure treats the p-values from n repeated random splits of one fixed user population as independent and identically distributed, even though A/A groups built from the same users overlap and the resamples are therefore correlated.","fun_headline_variants_meta":{"raw":{"variants":["Check p-value uniformity to verify A/B test validity","KS test on A/A resamples catches unreliable t-tests","A/A resampling plus KS test flags skewed p-values","Validate A/B tests by checking null p-values are uniform"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1394,"prompt_tokens":947,"completion_tokens":447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":563,"tokens_out":447,"duration_ms":4973,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:26:56.277444+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate user-level outcomes with a known skewed or zero-inflated distribution and a true effect of zero, run the proposed 5,000-resample A/A procedure, and compare KS flags against how often the stated 95% confidence intervals actually contain the true zero effect; if the KS test fails to flag metrics whose coverage is far from 95%, or flags metrics whose coverage is correct, the method's diagnostic value is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standard A/B-testing reference and the statement that p-values should be uniform under the null hypothesis (cited at p. 188)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the central limit theorem foundation that justifies treating the sample mean's distribution as normal in large samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies why the Mann-Whitney U-test is not a drop-in replacement for the t-test, because it tests a different null hypothesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the footnote about multiple-testing correction when interpreting Kolmogorov-Smirnov p-values across many metrics."}],"review_version":1}