Pith. sign in

REVIEW

Do more observations bring more information in rare events?

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.13671 v1 pith:F343NOX4 submitted 2025-06-16 stat.ME

Do more observations bring more information in rare events?

classification stat.ME
keywords eventsrareobservationspowersampletestboostedcomputational
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

It is generally believed that more observations provide more information. However, we observe that in the independence test for rare events, the power of the test is, surprisingly, determined by the number of rare events rather than the total sample size. Moreover, the correlations tend to shrink to zero even as the total sample size increases, as long as the proportion of rare events decreases. We demonstrate this phenomenon in both fixed and high-dimensional settings. To address these issues, we first rescale the covariances to account for the presence of rare events. We then propose a boosted procedure that uses only a small subset of non-rare events, yet achieves nearly the same power as using the full set of observations. As a result, computational complexity is significantly reduced. The theoretical properties, including asymptotic distribution and local power analysis, are carefully derived for both the rescaled statistic based on the full sample and the boosted test statistic based on subsampling. Furthermore, we extend the theory to multi-class rare events. Extensive simulations and real-world data analyses confirm the effectiveness and computational efficiency of the proposed approach.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.