{"id":"fd8a1e6a-b680-4d85-a5e4-b8b74ea5b872","arxiv_id":"2411.13542","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The Rényi Outlier Test generalizes the Cox-Kartsonaki Rényi transformation to be robust to the unknown number of outliers and to accept prior probabilities and effect-size weights.","lead":"A new statistical test, the Rényi Outlier Test, detects outlying p-values in large-vector multiple testing without needing to know exactly how many outliers to expect. The method also lets users fold in prior information about which p-values are likely to be outliers and how far out they might be.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The compensator in §2 assumes π and η are fixed, but the paper's 'estimate' framing invites data-dependent priors, which would break the i.i.d. exponential interarrival guarantee.","rationale":"The reader's weakest_assumption points to the same issue: π and η are treated as fixed, non-random, but the paper's language suggests they may be estimated. This is the most load-bearing concern because it directly threatens the validity of the test in a common usage pattern: users naturally estimate prior probabilities and effect sizes from the very p-values they are testing. If those estimates are plugged into equations (1)–(3), the time-change theorem no longer yields independent exponential interarrival times, so the null distribution of ρ_{K*} used to compute p-values is incorrect. The mathematical construction itself appears internally sound when π and η are truly fixed; the spline and Monte Carlo null distribution, while lacking error analysis, is a calibration detail rather than a fundamental validity issue. The absence of simulations is a deficiency in evidence but does not by itself identify a flaw in the argument. The prior measurability condition is explicitly asserted in §2 yet not reconciled with the 'estimate' framing, making it a concrete, potentially fatal gap. A simulation that estimates π and η from null data and checks the uniformity of the resulting p-values would settle whether the concern lands. Since the reader's CONDITIONAL verdict already requests guidance on this point, reinforcing it does not change the verdict.","tokens_in":3599,"tokens_out":14321,"duration_ms":150806,"concrete_test":"Simulate M = 10^5 null datasets of p p-values. On each dataset, estimate π and η from the same data (e.g., empirical Bayes marginal estimates of the outlier probability and of the conditional effect size from the p-values or their transformations), run the ROT as implemented in the renyi package, and collect the resulting p-value. Test the M p-values for uniformity with a Kolmogorov–Smirnov test and check the empirical Type I error at α = 0.05. If the KS test rejects with large margin or the Type I error exceeds 0.06, the fixed-prior assumption is breached and the claimed distributional guarantee fails. An analytical alternative is to re-derive the compensator when π_j and η_j are arbitrary functions of U_j and verify whether equation (3) remains a valid predictable compensator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in Section 2, equations (1)–(3). The compensator Λ(t) and the resulting i.i.d. unit exponential interarrival times X_j require π and η to be fixed, non-random quantities; the text says 'We understand η_j and π_j to be measurable with respect to F_t for all t (including t = −∞)', which effectively forces them to be constant. Yet the abstract and Section 1 describe η as 'an estimate of how far of an outlier a p-value might be' and π as prior probabilities, which in practice are often estimated from the same p-values being tested. If π or η are data-dependent, Z_j = η_j(−log U_j + log π_j) is no longer a scaled/truncated exponential, the point-process compensator (3) is not the correct compensator, and the Rényi representation of X_j as i.i.d. unit exponentials breaks down. Consequently the null distribution of ρ_{K*} in equation (5) would be miscalibrated, so the central claim that ROT yields valid tests while accounting for prior information is unsupported in the most natural data-dependent-prior setting. The paper gives no warning, no correction, and no consistency result for estimated π or η.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Rényi Outlier Test (ROT), a generalization of Cox and Kartsonaki's Rényi-transformation-based outlier test for p-values. The method transforms the ordered p-values with prior weights π (prior outlier probabilities) and η (effect-size weights), applies a time-change argument to obtain i.i.d. unit exponential interarrival times under the global null, and then aggregates evidence over a dyadic grid of candidate outlier counts up to K* = 2^{ceil(log2 K)} to avoid specifying the exact number of outliers. The null distribution of the resulting test statistic is approximated by Monte Carlo simulation with a spline fitted to the body and a line fitted to the tail, and the method is implemented in an R package named renyi. The paper claims that ROT maintains the computational speed and numerical precision of the original test while being 'much more robust' to uncertainty in the expected number of outliers, and that it can incorporate the two types of prior information described above.","tokens_in":3791,"tokens_out":7102,"duration_ms":78338,"significance":"If the claimed properties are correct, ROT addresses a practical limitation of the Cox–Kartsonaki test, namely its sensitivity to the a priori number of outliers k, and it extends the framework to include prior information. The mathematical derivation is plausible: it uses standard results (Kallenberg's time-change theorem and Rényi's representation of exponential order statistics), and the construction of \tilde{X}_{K*} appears internally consistent. The provision of the renyi R package and pre-calculated spline lookup tables is a concrete contribution. However, the paper contains no simulation study, no power comparison against existing methods, no real-data example, and no validation of the spline-based null distribution approximation in the extreme tail. Consequently, the central claims of robustness, numerical stability, and valid p-value calculation are unverified. The paper is therefore a promising methodological proposal rather than a fully supported statistical method.","major_comments":[{"comment":"The central claim that ROT is 'much more robust to uncertainty in the number of outliers expected a priori among the p-values' is not supported by any simulation, power comparison, or real-data analysis in the manuscript. The paper describes the test statistic (5) and its null distribution but reports no type I error rates, no power curves for different true values of k relative to the chosen bound K, and no comparison with Cox and Kartsonaki's original test, Higher Criticism, or Bonferroni/Holm procedures. Since this robustness property is the main contribution, the authors should provide a simulation study with calibration checks (e.g., empirical null size across choices of K and p) and power comparisons under sparse alternatives with k both smaller than and equal to K. Without such evidence, the abstract's claim is not established.","section":"Abstract and Section 3 (definition of ρ_{K*})"},{"comment":"The derivation of the i.i.d. unit exponential interarrival times assumes that π_j and η_j are fixed, non-random quantities; the text states that η_j and π_j are 'measurable with respect to F_t for all t (including t = -∞)', which effectively forces them to be constants. However, the abstract and the introduction describe η as 'an estimate of how far of an outlier a p-value might be' and π as prior probabilities. If these quantities are estimated from the same p-values U being tested, then Z_j = η_j(-log U_j + log π_j) is no longer a scaled and shifted exponential, the compensator (3) is not the correct compensator, and the Rényi representation of the interarrival times as i.i.d. unit exponentials breaks down. The paper should either state clearly that π and η must be specified independently of the data and should not be called estimates, or it should provide a theoretical result and numerical evidence for the behavior of the test when π and η are data-dependent. As written, the validity conditions of the test are ambiguous and potentially misleading.","section":"Section 2, Eqs. (1)–(3)"},{"comment":"The null distribution of ρ_{K*} is approximated by simulating the statistic, fitting a line to the log-linear tail, and fitting a cubic spline to the body, but the paper gives no details on the Monte Carlo sample size, the fitting error, the range of K* covered, or the accuracy of the approximation in the extreme tail (e.g., p-values around 10^{-10}). The abstract and introduction emphasize numerical stability for very small p-values as a key advantage, so if the tail approximation is inaccurate, the reported p-values may be badly miscalibrated precisely in the regime the method claims to handle. The authors should report validation of the approximation—for example, by comparing the spline-based p-values with direct Monte Carlo estimates at several extreme quantiles and by stating the simulation size used to fit the splines. The package should also make the lookup table and simulation procedure available for independent verification.","section":"Page 3, 'We used Monte Carlo simulation...'"},{"comment":"The transformation \tilde{X}_{K*} = -log(1 - I_{p-K*+1,K*}(exp(-sum_{j=K*}^p X_j/j))) is described by invoking Rényi's representation, but the paper does not demonstrate that this variable is a unit exponential independent of \tilde{X}_1,...,\tilde{X}_{K*-1}. The distributional claim follows from standard results, but the independence is not immediate from the displayed formula alone. A brief derivation or a precise statement of the relevant Rényi representation property would make the construction more transparent and easier for readers to verify.","section":"Eqs. (4)–(5)"}],"minor_comments":[{"comment":"The phrase 'two types of prior information that is common in modern applications' should read 'that are common'.","section":"Abstract"},{"comment":"The notation K* := 2⌈log2 K⌉ is ambiguous on the page; it should be written as 2^{⌈log2 K⌉} to make clear that K* is a power of two.","section":"Page 2, introduction"},{"comment":"The set I_k is defined as (1, 2, 4, ..., K*), but the subscript k is not defined; it should be I_{K*} to avoid confusion with the earlier k denoting the number of outliers.","section":"Eq. (5)"},{"comment":"The sentence 'We then test the outliers based on those exponential random variables' should be 'We then test for outliers' to avoid implying that the outliers themselves are known.","section":"Page 2, final paragraph"},{"comment":"The reference to 'Kallenberg (2021)' is to the third edition of 'Foundations of Modern Probability'; the specific theorem number (Theorem 15.15) is not provided in the reference list. Please verify that the theorem number is correct in that edition and provide the page or chapter if possible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is very short (four pages) and presents a method with a plausible derivation but no empirical validation. The main concern is the absence of any simulation or data analysis to support the robustness and calibration claims; a major revision with a thorough simulation study is necessary. The data-dependent-prior issue is also a real scope limitation that should be addressed by clear statements about the assumptions on π and η. I would not reject the paper because the underlying idea is sound and the package is a useful contribution, but the current manuscript is not yet ready for publication in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a genuine, mathematically careful generalization of Cox and Kartsonaki’s Rényi outlier test. It adds prior weights π and η and an omnibus max over dyadic candidate outlier counts, giving a test that plausibly holds its power when the number of outliers is only roughly known. The derivation leans on Kallenberg’s time-change theorem and Rényi’s representation, and the logic is sound on the page. The authors also ship an R package with precomputed spline approximations, so the computational story is plausible.\n\nBut the paper is almost entirely theoretical. There are no simulations, no power comparisons, and no real-data illustration anywhere. The abstract’s central claim—that ROT is “much more robust to uncertainty in the number of outliers”—is never quantified. We don’t see how the omnibus over dyadic k behaves compared to knowing k, nor how it degrades as K overestimates the true count. Similarly, the claimed value of π and η for sharpening the alternative is asserted, not demonstrated.\n\nThe bigger issue, which the stress-test flagged, is real: the compensator in (1)–(3) requires π and η to be fixed, non-random. The sentence “we understand η_j and π_j to be measurable with respect to F_t for all t (including t = −∞)” effectively forces them to be constant. But the abstract describes η as “an estimate of how far of an outlier a p-value might be” and π as prior probabilities—objects that in practice get estimated from the same p-values under test. If that happens, the i.i.d. exponential interarrival property breaks and the null distribution of ρ_{K*} is miscalibrated. The paper gives no warning, no correction, and no consistency result. This is a genuine gap, not a manufactured one.\n\nAlso minor: the null distribution is approximated by Monte Carlo plus a cubic spline fit to the body and a log-linear tail. There is no error analysis for that approximation, so the reported p-values inherit unknown error. That’s probably fixable with a supplementary analysis, but it’s worth asking.\n\nThis paper deserves a serious referee because the construction is novel and the math appears correct under stated assumptions. But the revision bar is real: add simulations, power comparisons, real-data examples, and explicitly discuss what happens when π or η are estimated. If you work on multiple testing or p-value combination, read it for the idea; don’t cite the robustness claim until the evidence is in.","headline":"Plausible math and a clean generalization of Cox–Kartsonaki, but the key robustness and utility claims have no supporting simulations or data-dependent-prior guidance.","tokens_in":4358,"tokens_out":2361,"would_cite":false,"duration_ms":23878,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62G10","62G30"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes the Rényi Outlier Test, a generalization of the original Rényi-transformation p-value outlier test that needs only a rough upper bound K on the number of outliers and can incorporate prior probabilities and effect-size…","keywords":["outlier test","p-value combination","Higher Criticism","sparse alternatives","global null","Rényi transformation","large-scale multiple testing"],"falsifier":"Take $p=10{,}000$ p-values all drawn independently from Uniform(0,1), fix $\\pi_j=\\eta_j=1$ and any upper bound $K$, run ROT, and collect the reported p-values over many replicates: under the claimed null they should be Uniform(0,1). Any systematic deviation larger than Monte Carlo error, especially for small $K^\\star$ where splines are fitted, would falsify the implementation; separately, simulating alternatives with exactly $k$ true outliers and comparing ROT at $K>k$ with the original test at various guessed $k$ would test the robustness claim.","tokens_in":3332,"feed_emoji":"🎯","tokens_out":12930,"duration_ms":119370,"temperature":0.7,"pith_summary":"The paper proposes the Rényi Outlier Test (ROT), a generalization of a recently proposed p-value outlier test based on the Rényi transformation. The original test is fast for large $p$ and numerically stable for very small p-values, but its power depends on specifying the exact number of outliers $k$ in advance; choosing $k$ too large dilutes the signal. ROT replaces the exact $k$ with a rough upper bound $K$ and maximizes over dyadic candidate counts up to $K^\\star = 2^{\\lceil \\log_2 K\\rceil}$, which the paper argues by construction and simulation retains power while avoiding the dilution problem. It also accepts priors $\\pi$ for the probability that a given p-value is outlying and weights $\\eta$ for its effect size, defaulting to neutral values when no prior information exists. If the claims hold, ROT gives large-scale multiple-testing analyses a fast, numerically stable outlier test that does not require knowing the number of true signals.","feed_headline":"P-value outlier test stays strong when outlier count is unknown","feed_subtitle":"Needs only a rough upper bound K, accepts prior probabilities and effect sizes, and comes with an R package.","key_machinery":"The central object is the generalized Rényi transformation with a compensator. For ordered values, the original map $\\rho(u)_j = j\\log(u_{j+1}/u_j)$ turns i.i.d. uniforms into independent exponentials; ROT replaces it by a weighted version $Z_j = \\eta_j(-\\log U_j + \\log\\pi_j)$ and uses the compensator $\\Lambda(t)=\\sum_j \\eta_j^{-1}(t\\wedge Z_j - t\\wedge \\zeta_j)$ to extract independent unit exponentials from the counting process. This is what lets the test carry prior information without destroying the null distribution. The second mechanism is the dyadic maximum: instead of testing one candidate $k$, the statistic takes the maximum over $i=1,2,4,\\dots,K^\\star$ of $-\\log(1-G_i(\\cdot))$, an omnibus test over possible outlier counts up to the upper bound. The third piece is the precomputed spline lookup table for the null distribution of $\\rho_{K^\\star}$, which makes the implementation fast and numerically stable.","core_discovery":"Under the global null that the $p$-values are i.i.d. Uniform(0,1), the paper defines a weighted Rényi transformation $Z_j = \\eta_j(-\\log U_j + \\log\\pi_j)$ and shows, via the compensator $\\Lambda(t)$, that the transformed counting process has unit-rate Poisson interarrival times $X_1,\\dots,X_p$, independent standard exponentials. A final exponential $\\tilde{X}_{K^\\star}$ is constructed from the largest order statistic through a Beta transform so that the whole set remains independent. The test statistic is $\\rho_{K^\\star} = \\max_{i \\in \\{1,2,4,\\dots,K^\\star\\}} -\\log(1 - G_i(\\sum_{j=1}^i \\tilde{X}_j))$, where $G_i$ is the Gamma($i$,1) CDF, and the paper precomputes its null distribution with splines. The central claim is that this statistic preserves the speed and numerical precision of the original Rényi test while being robust to uncertainty in the number of outliers, and that the spline-based null distribution is accurate enough for practical use.","pith_inferences":["The compensator construction is not tied to the specific choice of $\\eta$ and $\\pi$; the same Poisson-process argument would yield valid exponential interarrival times for other monotone transforms of p-values, so the ROT template could extend to other weighted or grouped p-value summaries.","Because the effect-size prior $\\eta_j \\propto E[\\beta_j^2 \\mid \\beta_j \\ne 0]$ is a natural object in regression, ROT could be plugged into large regression scans where per-variable effect-size estimates are available; the paper states the prior but does not run that application.","A dyadic maximum over $K^\\star$ is one omnibus choice; other combinations of the independent exponentials might give different power trade-offs, and the robustness claim suggests comparing them under overestimated $K$ as a useful follow-up."],"forward_implications":["ROT only requires a rough upper bound $K$ on the number of outlying p-values; overestimating $K$ no longer crushes power the way choosing $k$ too large does in the original test.","The test admits per-variable prior probabilities $\\pi$ and effect-size weights $\\eta$, so analyses with external information about which hypotheses are likely false can feed that information in; with no information, the defaults $\\pi_j=\\eta_j=1$ recover an unweighted robust Rényi test.","Reported p-values remain accurate down to very small values, matching the original test's numerical precision, because the null distribution is stored as precomputed splines rather than simulated at query time.","The renyi R package implements the procedure, so the method is available for immediate use on large p-value vectors."],"supporting_citations":[{"why":"Supplies the original Rényi-transformation outlier test that ROT generalizes, including the baseline speed, numerical stability, and sensitivity to the exact number of outliers.","marker":"Cox and Kartsonaki (2019)"},{"why":"Provides the theorem that the Rényi transformation maps i.i.d. uniforms to independent exponentials and the order-statistics representation used to build the final independent exponential.","marker":"Rényi (1953)"},{"why":"Supplies the point-process theorem used to show that the compensated counting process becomes a unit-rate Poisson process.","marker":"Kallenberg (2021)"},{"why":"Motivates the sparse-outlier testing regime by analyzing Higher Criticism, the family of tests ROT is intended to improve on in speed and flexibility.","marker":"Donoho and Jin (2004)"},{"why":"Provides the fast, numerically stable Higher Criticism implementation that ROT competes with and extends by admitting prior information.","marker":"Wang et al. (2024)"}],"fun_headline_variants":["Robust p-value outlier test for unknown outlier counts","Rényi Outlier Test handles uncertain outlier count and priors","Fast outlier test that stays accurate when outlier number is unknown","New outlier test robust to unknown outlier count, with R package","Outlier test that uses priors on probabilities and effect sizes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test's validity assumes the prior weights $\\pi$ and $\\eta$ are fixed in advance and not estimated from the same p-values being tested; if a user learns the weights from the data, the independent-exponential property that the p-value calculation relies on no longer follows.","fun_headline_variants_meta":{"raw":{"variants":["Robust p-value outlier test for unknown outlier counts","Rényi Outlier Test handles uncertain outlier count and priors","Fast outlier test that stays accurate when outlier number is unknown","New outlier test robust to unknown outlier count, with R package","Outlier test that uses priors on probabilities and effect sizes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3169,"prompt_tokens":952,"completion_tokens":2217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2133}},"tokens_in":568,"tokens_out":2217,"duration_ms":15136,"temperature":1.0,"reasoning_tokens":2133,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:18:10.118987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take $p=10{,}000$ p-values all drawn independently from Uniform(0,1), fix $\\pi_j=\\eta_j=1$ and any upper bound $K$, run ROT, and collect the reported p-values over many replicates: under the claimed null they should be Uniform(0,1). Any systematic deviation larger than Monte Carlo error, especially for small $K^\\star$ where splines are fitted, would falsify the implementation; separately, simulating alternatives with exactly $k$ true outliers and comparing ROT at $K>k$ with the original test at various guessed $k$ would test the robustness claim.","supporting_citations":[{"cited_title":"On the analysis of large numbers of p-values","cited_arxiv_id":null,"evidence_quote":"Supplies the original Rényi-transformation outlier test that ROT generalizes, including the baseline speed, numerical stability, and sensitivity to the exact number of outliers."},{"cited_title":"Foundations of Modern Probability","cited_arxiv_id":null,"evidence_quote":"Supplies the point-process theorem used to show that the compensated counting process becomes a unit-rate Poisson process."},{"cited_title":"Higher criticism for detecting sparse heterogeneous mixtures","cited_arxiv_id":null,"evidence_quote":"Motivates the sparse-outlier testing regime by analyzing Higher Criticism, the family of tests ROT is intended to improve on in speed and flexibility."},{"cited_title":"Accurate and ultra-efficient p-value calculation for higher criticism tests","cited_arxiv_id":null,"evidence_quote":"Provides the fast, numerically stable Higher Criticism implementation that ROT competes with and extends by admitting prior information."}],"review_version":1}