Pith. sign in

REVIEW 4 major objections 5 minor 7 references

The R\'enyi Outlier Test

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes the Rényi Outlier Test, a generalization of the original Rényi-transformation p-value outlier test that needs only a rough upper bound K on the number of outliers and can incorporate prior probabilities and effect-size…

desk verdict Plausible math and a clean generalization of Cox–Kartsonaki, but the key robustness and utility claims have no supporting simulations or data-dependent-prior guidance. read the letter →

arxiv 2411.13542 v1 pith:OJGX6MTX submitted 2024-11-20 stat.ME

Ryan Christ , Ira Hall , David Steinsaltz This is my paper · ORCID
classification stat.ME MSC 62F0362G1062G30
keywords outliertestp-valuecombinationHigherCriticismsparsealternativesglobalnullRényitransformationlarge-scalemultipletesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes the Rényi Outlier Test (ROT), a generalization of a recently proposed p-value outlier test based on the Rényi transformation. The original test is fast for large $p$ and numerically stable for very small p-values, but its power depends on specifying the exact number of outliers $k$ in advance; choosing $k$ too large dilutes the signal. ROT replaces the exact $k$ with a rough upper bound $K$ and maximizes over dyadic candidate counts up to $K^\star = 2^{\lceil \log_2 K\rceil}$, which the paper argues by construction and simulation retains power while avoiding the dilution problem. It also accepts priors $\pi$ for the probability that a given p-value is outlying and weights $\eta$ for its effect size, defaulting to neutral values when no prior information exists. If the claims hold, ROT gives large-scale multiple-testing analyses a fast, numerically stable outlier test that does not require knowing the number of true signals.

What carries the argument

The central object is the generalized Rényi transformation with a compensator. For ordered values, the original map $\rho(u)_j = j\log(u_{j+1}/u_j)$ turns i.i.d. uniforms into independent exponentials; ROT replaces it by a weighted version $Z_j = \eta_j(-\log U_j + \log\pi_j)$ and uses the compensator $\Lambda(t)=\sum_j \eta_j^{-1}(t\wedge Z_j - t\wedge \zeta_j)$ to extract independent unit exponentials from the counting process. This is what lets the test carry prior information without destroying the null distribution. The second mechanism is the dyadic maximum: instead of testing one candidate $k$, the statistic takes the maximum over $i=1,2,4,\dots,K^\star$ of $-\log(1-G_i(\cdot))$, an omnibus test over possible outlier counts up to the upper bound. The third piece is the precomputed spline lookup table for the null distribution of $\rho_{K^\star}$, which makes the implementation fast and numerically stable.

What would settle it

Take $p=10{,}000$ p-values all drawn independently from Uniform(0,1), fix $\pi_j=\eta_j=1$ and any upper bound $K$, run ROT, and collect the reported p-values over many replicates: under the claimed null they should be Uniform(0,1). Any systematic deviation larger than Monte Carlo error, especially for small $K^\star$ where splines are fitted, would falsify the implementation; separately, simulating alternatives with exactly $k$ true outliers and comparing ROT at $K>k$ with the original test at various guessed $k$ would test the robustness claim.

Watch

Extended reading notes

Core claim

Under the global null that the $p$-values are i.i.d. Uniform(0,1), the paper defines a weighted Rényi transformation $Z_j = \eta_j(-\log U_j + \log\pi_j)$ and shows, via the compensator $\Lambda(t)$, that the transformed counting process has unit-rate Poisson interarrival times $X_1,\dots,X_p$, independent standard exponentials. A final exponential $\tilde{X}_{K^\star}$ is constructed from the largest order statistic through a Beta transform so that the whole set remains independent. The test statistic is $\rho_{K^\star} = \max_{i \in \{1,2,4,\dots,K^\star\}} -\log(1 - G_i(\sum_{j=1}^i \tilde{X}_j))$, where $G_i$ is the Gamma($i$,1) CDF, and the paper precomputes its null distribution with splines. The central claim is that this statistic preserves the speed and numerical precision of the original Rényi test while being robust to uncertainty in the number of outliers, and that the spline-based null distribution is accurate enough for practical use.

Load-bearing premise

The test's validity assumes the prior weights $\pi$ and $\eta$ are fixed in advance and not estimated from the same p-values being tested; if a user learns the weights from the data, the independent-exponential property that the p-value calculation relies on no longer follows.

Editorial extensions

If this is right

  • ROT only requires a rough upper bound $K$ on the number of outlying p-values; overestimating $K$ no longer crushes power the way choosing $k$ too large does in the original test.
  • The test admits per-variable prior probabilities $\pi$ and effect-size weights $\eta$, so analyses with external information about which hypotheses are likely false can feed that information in; with no information, the defaults $\pi_j=\eta_j=1$ recover an unweighted robust Rényi test.
  • Reported p-values remain accurate down to very small values, matching the original test's numerical precision, because the null distribution is stored as precomputed splines rather than simulated at query time.
  • The renyi R package implements the procedure, so the method is available for immediate use on large p-value vectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The compensator construction is not tied to the specific choice of $\eta$ and $\pi$; the same Poisson-process argument would yield valid exponential interarrival times for other monotone transforms of p-values, so the ROT template could extend to other weighted or grouped p-value summaries.
  • Because the effect-size prior $\eta_j \propto E[\beta_j^2 \mid \beta_j \ne 0]$ is a natural object in regression, ROT could be plugged into large regression scans where per-variable effect-size estimates are available; the paper states the prior but does not run that application.
  • A dyadic maximum over $K^\star$ is one omnibus choice; other combinations of the independent exponentials might give different power trade-offs, and the robustness claim suggests comparing them under overestimated $K$ as a useful follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Rényi Outlier Test (ROT), a generalization of Cox and Kartsonaki's Rényi-transformation-based outlier test for p-values. The method transforms the ordered p-values with prior weights π (prior outlier probabilities) and η (effect-size weights), applies a time-change argument to obtain i.i.d. unit exponential interarrival times under the global null, and then aggregates evidence over a dyadic grid of candidate outlier counts up to K* = 2^{ceil(log2 K)} to avoid specifying the exact number of outliers. The null distribution of the resulting test statistic is approximated by Monte Carlo simulation with a spline fitted to the body and a line fitted to the tail, and the method is implemented in an R package named renyi. The paper claims that ROT maintains the computational speed and numerical precision of the original test while being 'much more robust' to uncertainty in the expected number of outliers, and that it can incorporate the two types of prior information described above.

Significance. If the claimed properties are correct, ROT addresses a practical limitation of the Cox–Kartsonaki test, namely its sensitivity to the a priori number of outliers k, and it extends the framework to include prior information. The mathematical derivation is plausible: it uses standard results (Kallenberg's time-change theorem and Rényi's representation of exponential order statistics), and the construction of ilde{X}_{K*} appears internally consistent. The provision of the renyi R package and pre-calculated spline lookup tables is a concrete contribution. However, the paper contains no simulation study, no power comparison against existing methods, no real-data example, and no validation of the spline-based null distribution approximation in the extreme tail. Consequently, the central claims of robustness, numerical stability, and valid p-value calculation are unverified. The paper is therefore a promising methodological proposal rather than a fully supported statistical method.

major comments (4)
  1. [Abstract and Section 3 (definition of ρ_{K*})] The central claim that ROT is 'much more robust to uncertainty in the number of outliers expected a priori among the p-values' is not supported by any simulation, power comparison, or real-data analysis in the manuscript. The paper describes the test statistic (5) and its null distribution but reports no type I error rates, no power curves for different true values of k relative to the chosen bound K, and no comparison with Cox and Kartsonaki's original test, Higher Criticism, or Bonferroni/Holm procedures. Since this robustness property is the main contribution, the authors should provide a simulation study with calibration checks (e.g., empirical null size across choices of K and p) and power comparisons under sparse alternatives with k both smaller than and equal to K. Without such evidence, the abstract's claim is not established.
  2. [Section 2, Eqs. (1)–(3)] The derivation of the i.i.d. unit exponential interarrival times assumes that π_j and η_j are fixed, non-random quantities; the text states that η_j and π_j are 'measurable with respect to F_t for all t (including t = -∞)', which effectively forces them to be constants. However, the abstract and the introduction describe η as 'an estimate of how far of an outlier a p-value might be' and π as prior probabilities. If these quantities are estimated from the same p-values U being tested, then Z_j = η_j(-log U_j + log π_j) is no longer a scaled and shifted exponential, the compensator (3) is not the correct compensator, and the Rényi representation of the interarrival times as i.i.d. unit exponentials breaks down. The paper should either state clearly that π and η must be specified independently of the data and should not be called estimates, or it should provide a theoretical result and numerical evidence for the behavior of the test when π and η are data-dependent. As written, the validity conditions of the test are ambiguous and potentially misleading.
  3. [Page 3, 'We used Monte Carlo simulation...'] The null distribution of ρ_{K*} is approximated by simulating the statistic, fitting a line to the log-linear tail, and fitting a cubic spline to the body, but the paper gives no details on the Monte Carlo sample size, the fitting error, the range of K* covered, or the accuracy of the approximation in the extreme tail (e.g., p-values around 10^{-10}). The abstract and introduction emphasize numerical stability for very small p-values as a key advantage, so if the tail approximation is inaccurate, the reported p-values may be badly miscalibrated precisely in the regime the method claims to handle. The authors should report validation of the approximation—for example, by comparing the spline-based p-values with direct Monte Carlo estimates at several extreme quantiles and by stating the simulation size used to fit the splines. The package should also make the lookup table and simulation procedure available for independent verification.
  4. [Eqs. (4)–(5)] The transformation ilde{X}_{K*} = -log(1 - I_{p-K*+1,K*}(exp(-sum_{j=K*}^p X_j/j))) is described by invoking Rényi's representation, but the paper does not demonstrate that this variable is a unit exponential independent of ilde{X}_1,..., ilde{X}_{K*-1}. The distributional claim follows from standard results, but the independence is not immediate from the displayed formula alone. A brief derivation or a precise statement of the relevant Rényi representation property would make the construction more transparent and easier for readers to verify.
minor comments (5)
  1. [Abstract] The phrase 'two types of prior information that is common in modern applications' should read 'that are common'.
  2. [Page 2, introduction] The notation K* := 2⌈log2 K⌉ is ambiguous on the page; it should be written as 2^{⌈log2 K⌉} to make clear that K* is a power of two.
  3. [Eq. (5)] The set I_k is defined as (1, 2, 4, ..., K*), but the subscript k is not defined; it should be I_{K*} to avoid confusion with the earlier k denoting the number of outliers.
  4. [Page 2, final paragraph] The sentence 'We then test the outliers based on those exponential random variables' should be 'We then test for outliers' to avoid implying that the outliers themselves are known.
  5. [References] The reference to 'Kallenberg (2021)' is to the third edition of 'Foundations of Modern Probability'; the specific theorem number (Theorem 15.15) is not provided in the reference list. Please verify that the theorem number is correct in that edition and provide the page or chapter if possible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the ROT derivation is self-contained, relying on external theorems and Monte Carlo simulation of the null distribution.

full rationale

The paper's derivation chain is not circular. The Rényi Outlier Test starts from the Rényi transformation and constructs a point process with compensator (3), then invokes Kallenberg's Theorem 15.15 to obtain i.i.d. unit exponential interarrival times under the global null. This is an external mathematical result applied to a fixed, pre-specified pair (π, η); no parameter is fitted to the p-values being tested. The robustified statistic ρ_{K*} in (5) is defined as a maximum over sums of these exponentials, and its null distribution is obtained by Monte Carlo simulation, with spline functions used only as a compressed numerical approximation of that simulated distribution for fast p-value computation. This is a legitimate computational reduction, not a fitted input renamed as a prediction. The paper contains no self-citations that carry the argument, and it does not invoke any uniqueness theorem from the authors' own prior work. The stated measurability assumption on π and η indicates they are treated as fixed or pre-visible, so any later use of data-dependent estimates would be an extension beyond the paper's stated assumptions rather than a circular derivation. No equation reduces by construction to its own inputs, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The free parameters and axioms listed are the core mathematical and computational dependencies. The spline coefficients are a fitted accessory that the implementation relies on. The axioms are either standard mathematical results or domain assumptions about exact uniformity and fixed priors. No invented entities are introduced.

free parameters (1)
  • spline coefficients for null distribution of ρ_{K*} = not reported
    The paper fits a line to the log-linear tail and a cubic spline to the body of the simulated null distribution for each K*. These coefficients underpin the p-values returned by the R package, but no coefficients, knot locations, or Monte Carlo sample sizes are reported.
assumptions (5)
  • standard math Kallenberg's Theorem 15.15 ensures N composed with the inverse compensator is a unit-rate Poisson process.
    Invoked in Section 2 (paragraph after equation (3)) to justify that the transformed interarrival times X_j are i.i.d. standard exponential under H0.
  • standard math Rényi's representation for the order statistics of independent exponential random variables.
    Used in Section 2 around equation (4) to combine the tail sum into the K*-th largest exponential and then to an independent unit exponential via the Beta CDF.
  • domain assumption Under the global null, the original p-values are exactly i.i.d. Uniform(0,1).
    Stated in Section 2: 'Note that if each U_j represents a p-value, they must be exactly uniform under the global null hypothesis, not sub-uniform or super-uniform.'
  • domain assumption The prior weights π and η are known, fixed constants, not estimated from the data.
    Introduced in Section 2 (equation (1)); the compensator construction assumes they are measurable with respect to the filtration and non-random. No guidance is given for estimated priors.
  • ad hoc to paper The Monte Carlo null distribution and the spline approximation accurately represent the distribution of ρ_{K*}, including the extreme tail.
    The paper fits a line to the log-linear tail and a cubic spline to the body from simulations without reporting error bounds or knot choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The R\'enyi Outlier Test." pith.science (2026). https://pith.science/paper/OJGX6MTX

@misc{pith2026241113542,
  author       = {Pith},
  title        = {Pith review of: The R\'enyi Outlier Test},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OJGX6MTX}},
  note         = {Machine review of arXiv:2411.13542}
}
abstract

Cox and Kartsonaki proposed a simple outlier test for a vector of p-values based on the R\'enyi transformation that is fast for large $p$ and numerically stable for very small p-values -- key properties for large data analysis. We propose and implement a generalization of this procedure we call the R\'enyi Outlier Test (ROT). This procedure maintains the key properties of the original but is much more robust to uncertainty in the number of outliers expected a priori among the p-values. The ROT can also account for two types of prior information that are common in modern data analysis. The first is the prior probability that a given p-value may be outlying. The second is an estimate of how far of an outlier a p-value might be, conditional on it being an outlier; in other words, an estimate of effect size. Using a series of pre-calculated spline functions, we provide a fast and numerically stable implementation of the ROT in our R package renyi.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

  1. [1]

    On the analysis of large numbers of p-values

    David R Cox and Christiana Kartsonaki. On the analysis of large numbers of p-values. International Statistical Review, 87 0 (3): 0 505--513, 2019

    work page 2019

  2. [2]

    Higher criticism for detecting sparse heterogeneous mixtures

    David Donoho and Jiashun Jin. Higher criticism for detecting sparse heterogeneous mixtures. The Annals of Statistics, 32 0 (3): 0 962--994, 2004

    work page 2004

  3. [3]

    Foundations of Modern Probability

    Olaf Kallenberg. Foundations of Modern Probability. Springer Verlag, third edition, 2021

    work page 2021

  4. [4]

    On the theory of order statistics

    Alfr \'e d R \'e nyi. On the theory of order statistics. Acta Math. Acad. Sci. Hung, 4 0 (2), 1953

    work page 1953

  5. [5]

    Accurate and ultra-efficient p-value calculation for higher criticism tests

    Wenjia Wang, Yusi Fang, Chung Chang, and George C Tseng. Accurate and ultra-efficient p-value calculation for higher criticism tests. Journal of Computational and Graphical Statistics, 33 0 (2): 0 463--476, 2024

    work page 2024

  6. [6]

    The general goodness-of-fit tests for correlated data

    Hong Zhang and Zheyang Wu. The general goodness-of-fit tests for correlated data. Computational Statistics & Data Analysis, 167: 0 107379, 2022

    work page 2022

  7. [7]

    Distributions and power of optimal signal-detection statistics in finite case

    Hong Zhang, Jiashun Jin, and Zheyang Wu. Distributions and power of optimal signal-detection statistics in finite case. IEEE Transactions on Signal Processing, 68: 0 1021--1033, 2020

    work page 2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.