Distributions and Statistical Power of Optimal Signal-Detection Methods In Finite Cases

In big data analysis for detecting rare and weak signals among $n$ features, some grouping-test methods such as Higher Criticism test (HC), Berk-Jones test (B-J), and $\phi$-divergence test share the similar asymptotical optimality when $n \rightarrow \infty$. However, in practical data analysis $n$ is frequently small and moderately large at most. In order to properly apply these optimal tests and wisely choose them for practical studies, it is important to know how to get the p-values and statistical power of them. To address this problem in an even broader context, this paper provides analytical solutions for a general family of goodness-of-fit (GOF) tests, which covers these optimal tests. For any given i.i.d. and continuous distributions of the input test statistics of the $n$ features, both p-value and statistical power of such a GOF test can be calculated. By calculation we compared the finite-sample performances of asymptotically optimal tests under the normal mixture alternative. Results show that HC is the best choice when signals are rare, while B-J is more robust over various signal patterns. In the application to a real genome-wide association study, results illustrate that the p-value calculation works well, and the optimal tests have potentials for detecting novel disease genes with weak genetic effects. The calculations have been implemented in an R package SetTest and published on the CRAN.