An asymptotically optimal transform of Pearson's correlation statistic
Pith reviewed 2026-05-24 15:11 UTC · model grok-4.3
The pith
For any correlation-parametrized model and any significance level, a transform of Pearson's R exists that removes the leading normal approximation error for all values of ρ.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
It is shown that for any correlation-parametrized model of dependence and any given significance level α∈(0,1), there is an asymptotically optimal transform of Pearson's correlation statistic R, for which the generally leading error term for the normal approximation vanishes for all values ρ∈(-1,1) of the correlation coefficient. This general result is then applied to the bivariate normal (BVN) model of dependence and to the SquareV model. In the BVN model, Pearson's R turns out to be asymptotically optimal for a rather unusual significance level α≈0.240, whereas Fisher's transform RF of R is asymptotically optimal for the limit significance level α=0. In the SquareV model, Pearson's R is 0.
What carries the argument
The asymptotically optimal transform of Pearson's correlation statistic R, chosen so that the leading error term in its asymptotic expansion under the normal approximation is canceled for every ρ.
If this is right
- In the bivariate normal model, the raw Pearson R is asymptotically optimal only for α≈0.240 while Fisher's transform is optimal only in the limit α→0.
- In the SquareV model, the raw Pearson R is asymptotically optimal for α≈0.159 while Fisher's transform is not optimal for any α in [0,1].
- The transform that is optimal for one fixed α outperforms both the raw R and Fisher's transform over wide intervals of significance levels that include the chosen α.
- For sample sizes n≥100 and significance levels α=0.01 or 0.05, the asymptotically optimal transform outperforms both R and Fisher's transform in simulations under both the bivariate normal and SquareV models.
Where Pith is reading between the lines
- The same cancellation technique could be applied to other test statistics whose Edgeworth expansions are known, yielding custom transforms tuned to common α values.
- For routine use, the optimal transforms for the standard levels 0.01, 0.05 and 0.10 could be tabulated once per model and then applied in software without recomputation.
- The advantage of the optimal transform shrinks as n grows large, because every reasonable transform converges to the same normal limit; the practical gain is therefore largest in the moderate-sample regime.
- If the dependence model is extended beyond a single correlation parameter, analogous transforms might still exist provided the leading error term remains a function of the parameters that can be canceled.
Load-bearing premise
The dependence structure must be fully determined by the single parameter ρ, and the distribution of R must admit an asymptotic expansion in which the leading error term can be identified and removed by a suitable function of R.
What would settle it
An explicit computation, for some correlation-parametrized model, of the coefficient of the leading error term in the expansion of R showing that this coefficient cannot be made to vanish for all ρ by any differentiable transform of R.
read the original abstract
It is shown that for any correlation-parametrized model of dependence and any given significance level $\alpha\in(0,1)$, there is an asymptotically optimal transform of Pearson's correlation statistic $R$, for which the generally leading error term for the normal approximation vanishes for all values $\rho\in(-1,1)$ of the correlation coefficient. This general result is then applied to the bivariate normal (BVN) model of dependence and to what is referred to in this paper as the SquareV model. In the BVN model, Pearson's $R$ turns out to be asymptotically optimal for a rather unusual significance level $\alpha\approx0.240$, whereas Fisher's transform $R_F$ of $R$ is asymptotically optimal for the limit significance level $\alpha=0$. In the SquareV model, Pearson's $R$ is asymptotically optimal for a still rather high significance level $\alpha\approx0.159$, whereas Fisher's transform $R_F$ of $R$ is not asymptotically optimal for any $\alpha\in[0,1]$. Moreover, it is shown that in both the BVN model and the SquareV model, the transform optimal for a given value of $\alpha$ is in fact asymptotically better than $R$ and $R_F$ in wide ranges of values of the significance level, including $\alpha$ itself. Extensive computer simulations for the BVN and SquareV models of dependence are presented, which suggest that, for sample sizes $n\ge100$ and significance levels $\alpha\in\{0.01,0.05\}$, the mentioned asymptotically optimal transform of $R$ generally outperforms both Pearson's $R$ and Fisher's transform $R_F$ of $R$, the latter appearing generally much inferior to both $R$ and the asymptotically optimal transform of $R$ in the SquareV model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript claims that for any correlation-parametrized model of dependence and any fixed significance level α ∈ (0,1), there exists a transform g_α of Pearson's R such that the leading error term in the normal approximation to the distribution of g_α(R) vanishes simultaneously for all ρ ∈ (-1,1). The result is specialized to the bivariate normal and SquareV models, identifying particular α values at which R or Fisher's RF is asymptotically optimal, and supported by simulations indicating that the model-specific optimal transform outperforms both R and RF for n ≥ 100 at α = 0.01 and 0.05.
Significance. If the general existence result can be established under explicit regularity conditions, the work would supply a systematic way to remove the leading O(n^{-1/2}) (or analogous) error in normal approximations to transformed correlation statistics across a broad class of dependence models. The concrete applications to BVN and SquareV together with the reported simulation comparisons would then constitute a useful contribution to the literature on refined asymptotic approximations for dependence measures.
major comments (2)
- [Abstract] Abstract (general existence statement): the claim that a single ρ-independent transform g_α cancels the leading error term for every correlation-parametrized model presupposes both the existence of a uniform asymptotic expansion for R whose leading term has an explicit, separable dependence on ρ and the quantile, and the solvability of the resulting functional equation for g_α. No regularity conditions (e.g., moment assumptions, smoothness of the joint density, uniformity in ρ) are stated; if the leading term contains irreducible model-specific components, the general assertion fails even though it may hold for the two examined models.
- [Applications to BVN and SquareV] Application to BVN and SquareV models: the paper asserts that the optimal transform for a given α is asymptotically superior to both R and RF over wide ranges of α, including the target α itself. The supporting derivation of the error-term cancellation and the explicit form of the optimal g_α should be checked for internal consistency with the model-specific expansions; any hidden dependence on fitted parameters would undermine the parameter-free character of the optimality claim.
minor comments (2)
- The simulation study is described as 'extensive' yet the abstract provides no information on the number of Monte Carlo replications, the precise numerical construction of the optimal transform, or measures of variability; these details are needed to evaluate whether the reported outperformance is statistically reliable.
- Notation for the SquareV model should be introduced with an explicit definition or reference in the main text rather than only in the abstract.
Simulated Author's Rebuttal
We thank the referee for the thoughtful and detailed report. We address the major comments below and will incorporate clarifications in a revised version where appropriate.
read point-by-point responses
-
Referee: [Abstract] Abstract (general existence statement): the claim that a single ρ-independent transform g_α cancels the leading error term for every correlation-parametrized model presupposes both the existence of a uniform asymptotic expansion for R whose leading term has an explicit, separable dependence on ρ and the quantile, and the solvability of the resulting functional equation for g_α. No regularity conditions (e.g., moment assumptions, smoothness of the joint density, uniformity in ρ) are stated; if the leading term contains irreducible model-specific components, the general assertion fails even though it may hold for the two examined models.
Authors: The general result is derived under the assumption that the correlation statistic R admits a uniform Edgeworth expansion in which the leading O(n^{-1/2}) term factors into a model-independent part depending on the quantile and a ρ-dependent coefficient that is separable. This holds for standard correlation-parametrized models with sufficient smoothness and moment conditions. We acknowledge that these conditions were not explicitly listed in the abstract or introduction. In the revision, we will add a dedicated subsection detailing the regularity conditions required for the existence result, including finite fourth moments, continuous differentiability of the joint density, and uniformity of the expansion over ρ in compact subsets of (-1,1). With these, the functional equation for g_α is solvable independently of the specific model. revision: yes
-
Referee: [Applications to BVN and SquareV] Application to BVN and SquareV models: the paper asserts that the optimal transform for a given α is asymptotically superior to both R and RF over wide ranges of α, including the target α itself. The supporting derivation of the error-term cancellation and the explicit form of the optimal g_α should be checked for internal consistency with the model-specific expansions; any hidden dependence on fitted parameters would undermine the parameter-free character of the optimality claim.
Authors: The model-specific expansions for BVN and SquareV are derived directly from the joint distributions without any parameter estimation in the transform itself. The optimal g_α is a fixed function depending only on α and the model structure, not on data or fitted ρ. We have re-examined the derivations and confirm internal consistency: the cancellation occurs exactly at the leading term for the chosen α, and the superiority holds asymptotically for ranges of α as stated. There is no hidden dependence on fitted parameters; the transforms are non-parametric in application. We can add a remark clarifying this in the revision if needed. revision: partial
Circularity Check
No circularity: existence proof via asymptotic expansion is self-contained
full rationale
The paper derives an existence result for an asymptotically optimal transform g_α(R) by identifying the leading error term in the normal approximation to the distribution of R and solving for a ρ-independent correction that cancels it. This is a direct analytic construction from the assumed expansion form; no parameter is fitted to data and then relabeled as a prediction, no self-citation supplies a load-bearing uniqueness theorem, and the result is not obtained by renaming a known empirical pattern. The two concrete models (BVN, SquareV) are used only to illustrate the general theorem, not to define it. The derivation therefore does not reduce to its own inputs by construction.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The distribution of the correlation statistic admits an asymptotic expansion whose leading error term can be canceled by a transform.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.