REVIEW 4 major objections 6 minor 30 references
Online activity prediction via generalized Indian buffet process models
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that stable beta-scaled process priors produce closed-form, empirically calibrated forecasts of new users, total triggers, and time to target sample size in online A/B tests, with gains concentrated at short pilot horizons.
desk verdict Useful closed-form Bayesian nonparametric forecasts for A/B testing, but a notation slip in the NB-SSP posterior and missing code need fixing before this is publishable as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the stable beta-scaled process (SBSP) prior, a completely random measure on user propensities built by scaling an alpha-stable process so it has a beta-type distribution on its largest jump; it induces power-law behavior and is analytically tractable under trait-process likelihoods. The load-bearing identities are Theorem 3.4's posterior decomposition into beta-distributed observed atoms plus an independent completely random measure, Corollary 3.5's specialization, Propositions 3.6 and 3.7 giving negative-binomial predictives for new users and trigger sums, and Theorem 3.8's truncated-geometric representation that enables exact simulation and inversion bands for the days-to-target quantity $D_M$.
What would settle it
Take a cohort of users observed for, say, 60 days and split their daily re-trigger counts by time since first trigger; fit NB-SSP on days 1-7 and compare forecast intervals with realized counts for users triggered recently versus long ago. If realized counts for recently triggered users systematically exceed the intervals while older users fall below, the constant-propensity exchangeability assumption is contradicted.
Extended reading notes
Core claim
The paper argues that placing a stable beta-scaled process (SBSP) prior on user propensities, rather than a beta prior or a beta-process prior, turns user-engagement forecasting into a collection of closed-form posterior and predictive statements. For Bernoulli daily activity (Be-SSP), negative binomial re-trigger counts (NB-SSP), and truncated-geometric first-trigger times (TG-SSP), the posterior of the underlying random measure splits into observed atoms with beta posteriors and an unobserved completely random measure whose largest-jump transform is Gamma; the number of newly triggering users follows a negative binomial distribution with a success probability determined by a beta-function contrast, and total triggers and time-to-threshold follow from the same calculus. The authors then fit hyperparameters by maximum marginal likelihood or curve fitting, and report improved accuracy on simulated data, 450+ public benchmark evaluations, and 1,774 production A/B tests, with the largest gains when only a few pilot days are available. If correct, the contribution is practical: experimenters can plan A/B test duration, forecast new-user volume and total trigger load, and quantify uncertainty without MCMC.
Load-bearing premise
Each user's day-to-day activity is independent and identically distributed given a fixed, user-specific propensity drawn from the prior; if real users' engagement changes after their first trigger or over time, the closed-form forecasts are misspecified.
Editorial extensions
If this is right
- Experimenters can forecast new-user counts and total trigger counts directly from closed-form expressions, without MCMC or variational inference.
- The method uses only a few pilot days (as few as $D_0 \le 7$ in the experiments) and still competes favorably with existing estimators in retrospective tests on 1,774 production A/B tests.
- The same posterior machinery yields an exact sampler and heuristic inversion intervals for the number of days needed to reach a target sample size.
- Because both coarse binary activity and granular count activity fit in the same framework, one model family covers the two common data formats for user engagement.
- The empirical Bayes strategy, via maximum marginal likelihood or curve fitting, removes the need for costly per-experiment prior elicitation.
Reading between the lines
- If non-stationary re-trigger behavior is common in real experiments, the exchangeability assumption will make NB-SSP intervals under-cover; a natural extension is a prior that lets propensity change with time since first trigger.
- The paper's own DG2 simulation shows TG-SSP degrades less than Be-SSP under recency effects, which suggests practitioners with coarse data should prefer TG-SSP when they suspect engagement decreases after the first trigger.
- Because the SBSP prior is distribution-free over the unknown mixing distribution, the same closed-form identities could be adapted to other sparse count forecasting tasks such as customer retention or rare-event monitoring.
- A testable extension is to compare NB-SSP forecasts against a model that includes arm-specific re-trigger rates, since the discussion notes treatment may affect re-trigger rates but not first-trigger times.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Bayesian nonparametric trait-process models for forecasting user engagement in online A/B tests. Three likelihood models are considered—Bernoulli (Be-SSP), negative binomial (NB-SSP), and truncated geometric (TG-SSP)—under a stable beta-scaled process prior on user-specific propensities. The authors derive posterior and predictive distributions in closed form, propose empirical Bayes hyperparameter estimation via maximum marginal likelihood or curve fitting, and evaluate the method on synthetic data, three public datasets, and 1,774 proprietary experiments. The main claimed contributions are closed-form forecasts of new users, total triggers, and time-to-target sample size without MCMC, with improved accuracy over competitors especially under short pilot periods.
Significance. If the derivations are correct, the paper offers a practically important advance: it makes a rich Bayesian nonparametric model computationally usable at industrial scale, with closed-form estimators for quantities that are typically obtained by simulation. The use of 1,774 proprietary A/B tests and the connection to the unseen-feature literature give the empirical claims unusually broad support. The paper also builds on and extends the Poisson partition calculus of James (2017) and Camerlenghi et al. (2022), and the accompanying generative schemes are useful for readers. However, the central technical contribution is currently undermined by an apparent inconsistency in the stated posterior density (Theorem 3.4) and by an ambiguous definition of the sufficient statistic m_n in the negative binomial model; these issues need to be resolved before the closed-form claims can be trusted.
major comments (4)
- [Theorem 3.4, Section 3.3] The posterior density of θ*_n is stated as proportional to (1−π_G(θ))^{D−m_n} θ^{−1−α} ∏_{d∈B_n} G(A_{d,n}; θ). Since B_n is the set of days with positive activity, the likelihood contribution from the remaining D−m_n days should be π_G(θ)^{D−m_n}, not (1−π_G(θ))^{D−m_n}; the factor (1−π_G(θ)) is the probability of activity, not inactivity. This is not a cosmetic issue: for Be-SSP with D=2 and a user active on one day, the printed theorem gives a density proportional to θ^{1−α} (i.e., Beta(2−α, 1)), while Corollary 3.5 and the generative scheme S3.1 give Beta(1−α, 2). Because Theorem 3.4 is the template for all posterior and predictive results, this factor must be corrected and the subsequent derivations rechecked.
- [Section 3.3 vs. Theorem 4.1] The symbol m_n is defined in Section 3.3 as the cardinality of B_n, the number of days user n is active, but Theorem 4.1 defines m_n = Σ_{d=1}^{D_0} A_{d,n}, the total trigger count. These coincide only when each user triggers at most once per day. The negative binomial case is especially affected: Corollary 3.5 states θ*_n ∼ Beta(m_n − α, rD + 1), which is consistent with m_n being the total trigger count, not the number of active days. A reader cannot tell which definition was used in the simulation studies, and with the Section 3.3 definition the claimed NB-SSP posterior would be misspecified even under the true data-generating model. The notation must be made consistent and the definition used in each result stated explicitly.
- [Abstract and Section 5.2] The abstract claims that the models show 'improved accuracy' in all settings compared with state-of-the-art competitors. However, the paper's own Zipf simulation (Section 5.2, Figure 6) concludes that Be-SSP is only 'competitive' with SSP and IBP, not uniformly better. This is a mismatch between the global claim and the reported evidence. The empirical claims should be restated to distinguish settings where the method improves on competitors from settings where it is comparable.
- [Section 5.1.1, DG2] The simulation study DG2 shows that Be-SSP degrades substantially when the exchangeability assumption is violated by post-first-trigger decay in activity, with TG-SSP becoming the better model. This is a useful robustness caveat, but it also means that the paper's general claim that the method is accurate under real-world re-trigger behavior needs a supporting empirical argument beyond the proprietary-data summary, especially since the model assumes constant user-specific propensity across days. I would like to see this limitation discussed explicitly in the main text rather than only in the simulation description.
minor comments (6)
- [Equation (1)] The second condition in the definition of U(D1)_D0 appears to read Σ_{d=1}^{D_0} A_{d,n} > 0, which contradicts the intended meaning of a previously unseen user triggering during the follow-up period; it should presumably be Σ_{d=1}^{D_1} A_{D_0+d,n} > 0.
- [Keywords] The keyword list contains the typo 'user prdiction'; it should be 'user prediction'.
- [Figure 4] The caption refers to 'BM' and 'GM' without defining them; these appear to be the Be-SSP and TG-SSP estimators, but the abbreviations are not introduced in the text or caption.
- [Section 5.2 and Figure 6] The text says D0 = 5 days are retained for training, while the caption of Figure 6 says D0 = 10; the numbers should be reconciled.
- [Section 6] There are several typographical errors in this section, including 'dasetest', 'NB-SB-SP', and 'usres'; a proofreading pass is needed.
- [Reproducibility] No code or data repository is provided for the simulations and real-data analyses, despite the paper's emphasis on closed-form, scalable computation; for an applied statistics paper this would substantially aid verification.
Circularity Check
No significant circularity: the posterior and predictive formulas are derived from external random-measure results (James 2017; Camerlenghi et al. 2022), and the empirical claims are evaluated on held-out follow-up periods.
full rationale
The paper's central derivation chain is not circular. Theorem 3.4 is proved in the supplement by combining the general trait-allocation posterior of James (2017) with the stable beta-scaled process representation in Lemma S2.1, and Corollary 3.5 plus Propositions 3.6-3.8 are algebraic consequences of that theorem. These expressions are not obtained by fitting the target predictive quantity; e.g., the estimator for new users, U-hat in eq. (10), is a closed-form posterior expectation under the SBSP prior, not a fitted parameter renamed as a prediction. The empirical Bayes hyperparameter estimation in Section 4 uses either the marginal likelihood (12) or a pilot-trajectory regression (13), but the reported accuracy is always measured on future days D0+1,...,D0+D1 that are not used in fitting; this is genuine held-out prediction. The same-model simulation in Section 5.1.1 is a self-consistency check rather than external validation, but it does not make any predictive quantity equal to its own input by construction, and the paper supplements it with Zipf-distributed data and real proprietary/ASOS benchmarks. The paper does rely on the SBSP prior from Camerlenghi et al. (2022), where two of the present authors are coauthors; however, that citation supplies an externally published, parameter-free construction with stated assumptions, and the present derivation does not reduce to the citation alone. The NB-SSP ambiguity noted by the skeptic, concerning whether m_n is the number of active days or the total trigger count, is a real mathematical consistency concern about Section 3.3 versus Theorem 4.1, but it is a correctness risk, not evidence that a claimed prediction reduces to a fitted input by construction. No circular step satisfying the required quote-and-reduction standard was found.
Assumptions & free parameters
free parameters (6)
- alpha =
estimated per experiment via Eq. (12) or Eq. (13)
- c =
estimated per experiment via Eq. (12) or Eq. (13)
- beta =
estimated per experiment via Eq. (12) or Eq. (13)
- r =
estimated via Eq. (12) for NB-SSP; fixed to 1 for Be-SSP and TG-SSP
- d0 (curve fitting anchor) =
1
- Dup (truncation horizon for D_M) =
3 times the point estimate of D_M
assumptions (4)
- domain assumption Given the directing random measure, scores A_{d,n} are independent across days and users and identically distributed per user (trait process exchangeability).
- standard math The prior for the directing measure is a stable beta-scaled process (SBSP), whose existence and Levy representation are taken from Camerlenghi et al. (2022).
- standard math Posterior representation for trait processes with CRM priors (Poisson partition calculus) from James (2017) is valid as used.
- domain assumption Hyperparameters can be estimated from the pilot by maximizing marginal likelihood or curve fitting, and plugging point estimates into posterior predictives remains calibrated enough for experiment planning.
Cite this review
Pith. "Pith review of Online activity prediction via generalized Indian buffet process models." pith.science (2026). https://pith.science/paper/BHKGP3GT
@misc{pith2026250519643,
author = {Pith},
title = {Pith review of: Online activity prediction via generalized Indian buffet process models},
year = {2026},
howpublished = {\url{https://pith.science/paper/BHKGP3GT}},
note = {Machine review of arXiv:2505.19643}
}
read the original abstract
Online A/B tests are the standard tool for data-driven decision-making at scale. Among the design choices with the largest impact on statistical power is the triggering mechanism: how many users to expose and for how long. This often requires forecasting user engagement, i.e., whether enough users will trigger, and when a target participation level will be reached, from limited pilot data. We introduce a Bayesian nonparametric model for predicting both new-user counts and total triggers, accommodating the heavy-tailed engagement patterns typical of web experiments. All predictive quantities can be computed without intensive numerical procedures such as MCMC or variational inference. We evaluate on three public datasets (over 450 public benchmark evaluations) and 1,774 proprietary A/B tests. In all the settings, our models show improved accuracy in forecasting new users, total triggers, and time to reach a target sample size compared with state-ofthe-art competitors, especially when only a few pilot days are observed.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Beraha, M. and S. Favaro (2023). Transform-scaled process priors for trait allocations in B ayesian nonparametrics. arXiv preprint arXiv:2303.17844\/
work page Pith review arXiv 2023
-
[2]
Broderick, T., M. I. Jordan, and J. Pitman (2012). Beta Processes, Stick-Breaking and Power Laws . Bayesian Analysis\/ 7\/ (2), 439 -- 476
work page 2012
-
[3]
Burnham, K. P. and W. S. Overton (1979). Robust estimation of population size when capture probabilities vary among animals. Ecology\/ 60\/ (5), 927--936
work page 1979
- [4]
-
[5]
Campbell, T., D. Cai, and T. Broderick (2018). Exchangeable trait allocations . Electronic Journal of Statistics\/ 12\/ (2), 2290 -- 2322
work page 2018
-
[6]
Carlin, B. P. and T. A. Louis (2000). Empirical B ayes: Past, present and future. Journal of the American Statistical Association\/ 95\/ (452), 1286--1289
work page 2000
- [7]
-
[8]
Deng, A. (2015). Objective B ayesian two sample hypothesis testing for online controlled experiments. In Proceedings of the 24th International Conference on World Wide Web , pp.\ 923--928
work page 2015
Show all 30 references
-
[9]
Efron, B. and R. Thisted (1976). Estimating the number of unseen species: How many words did S hakespeare know? Biometrika\/ 63\/ (3), 435--447
1976
-
[10]
Ferguson, T. S. and M. J. Klass (1972). A Representation of Independent Increment Processes without Gaussian Components . The Annals of Mathematical Statistics\/ 43\/ (5), 1634 -- 1643
1972
-
[11]
Good, I. J. (1953). The population frequencies of species and the estimation of population parameters. Biometrika\/ 40\/ (3-4), 237--264
1953
-
[12]
Good, I. J. and G. H. Toulmin (1956). The number of new species, and the increase in population coverage, when a sample is increased. Biometrika\/ 43\/ (1-2), 45--63
1956
-
[13]
Gravel, S. (2014). Predicting discovery rates of genomic features. Genetics\/ 197\/ (2), 601--610
2014
-
[14]
Griffiths, T. L. and Z. Ghahramani (2011). The I ndian buffet process: An introduction and review. Journal of Machine Learning Research\/ 12\/ (32), 1185--1224
2011
-
[15]
Kohavi, D
Gupta, S., R. Kohavi, D. Tang, Y. Xu, R. Andersen, E. Bakshy, N. Cardin, S. Chandran, N. Chen, D. Coey, et al. (2019). Top challenges from the first practical online controlled experiments summit. ACM SIGKDD Explorations Newsletter\/ 21\/ (1), 20--35
2019
-
[16]
Lange, and N
Ionita-Laza, I., C. Lange, and N. M. Laird (2009). Estimating the number of unseen variants in the human genome. Proceedings of the National Academy of Sciences\/ 106\/ (13), 5008--5013
2009
-
[17]
James, L. F. (2017). Bayesian P oisson calculus for latent feature modeling via generalized I ndian Buffet Process priors . The Annals of Statistics\/ 45\/ (5), 2016 -- 2045
2017
-
[18]
James, L. F., P. Orbanz, and Y. W. Teh (2015). Scaled subordinators and generalizations of the I ndian buffet process. arXiv: Probability\/
2015
-
[19]
Kingman, J. F. C. (1967). Completely random measures. Pacific Journal of Mathematics\/ 21\/ (1), 59 -- 78
1967
-
[20]
Kohavi, R., A. Deng, B. Frasca, T. Walker, Y. Xu, and N. Pohlmann (2013). Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pp.\ 1168--1176
2013
-
[21]
Liu, C. H. B., \^A . Cardoso, P. Couturier, and E. J. McCoy (2021). Datasets for online controlled experiments. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)
2021
-
[22]
Camerlenghi, S
Masoero, L., F. Camerlenghi, S. Favaro, and T. Broderick (2018). Posterior representations of hierarchical completely random measures in trait allocation models. In NeurIPS Workshop on All of B ayesian Nonparametrics
2018
-
[23]
Camerlenghi, S
Masoero, L., F. Camerlenghi, S. Favaro, and T. Broderick (2022). More for less: predicting and maximizing genomic variant discovery via B ayesian nonparametrics. Biometrika\/ 109\/ (1), 17--32
2022
-
[24]
Quenouille, M. H. (1956). Notes on bias in estimation. Biometrika\/ 43\/ (3/4), 353--360
1956
-
[25]
Richardson, T. S., Y. Liu, J. McQueen, and D. Hains (2022, 28--30 Mar). A B ayesian model for online activity sample sizes. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera (Eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , Volu...
2022
-
[26]
Storn, R. and K. Price (1997). Differential evolution--a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization\/ 11 , 341--359
1997
-
[27]
Teh, Y. and D. Gorur (2009). Indian buffet processes with power-law behavior. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Eds.), Advances in Neural Information Processing Systems , Volume 22. Curran Associates, Inc
2009
-
[28]
Tukey, J. (1958). Bias and confidence in not quite large samples. Ann. Math. Statist.\/ 29 , 614
1958
-
[29]
Wan, R., Y. Liu, J. McQueen, D. Hains, and R. Song (2023). Experimentation platforms meet reinforcement learning: B ayesian sequential decision-making for continuous monitoring. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pp.\ 5016--5027
2023
-
[30]
Valiant, P
Zou, J., G. Valiant, P. Valiant, K. Karczewski, S. O. Chan, K. Samocha, M. Lek, S. Sunyaev, M. Daly, and D. G. MacArthur (2016). Quantifying unobserved protein-coding variants in human populations provides a roadmap for large-scale sequencing projects. Nature communications\/ ...
2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.