Pith. sign in

REVIEW 3 major objections 5 minor 76 references

This paper introduces AGC and CMA, a pair of asymmetric measures that reduce to AUC for binary outcomes and to Spearman's rho for continuous pairs, with a unified asymptotic theory and a DeLong-type comparison test.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:55 UTC pith:I2ARKZ3K

load-bearing objection A genuinely useful unification of AUC and Spearman's rho, but the asymptotic inference leans on an unproved imported result, one theorem is false as stated, and the LLM significance claims look overblown. the 3 major comments →

arxiv 2510.17994 v2 pith:I2ARKZ3K submitted 2025-10-20 stat.ME

Assessing Monotone Dependence: Area Under the Curve Meets Rank Correlation

Eva-Maria Walz , Andreas Eberl , Tilmann Gneiting This is my paper
classification stat.ME MSC 62H2062G2062G10
keywords asymmetric grade correlationcoefficient of monotone associationarea under the curveSpearman's rhorank correlationmonotone dependenceDeLong testmid distribution function
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces two closely related asymmetric measures of monotone dependence — AGC and its affine transform CMA — and proves they unify two classical tools that were previously treated separately. When the outcome Y is binary, CMA equals the area under the ROC curve (AUC); when both variables are continuous, AGC equals Spearman's rank correlation. A third identity expresses CMA for continuous Y as a weighted average of AUC values over all binary thresholdings of Y. The measures also fix a known defect of symmetric rank correlations: AGC reaches its maximum of 1 if and only if Y is a monotone function of X, even when Y is discrete. The paper supplies central limit theorems for the plug-in estimators and a generalization of the classical DeLong test for comparing predictors on a shared outcome.

Core claim

The central claim is that AGC(X,Y) = cov(F̄(X),Ḡ(Y))/var(Ḡ(Y)) and CMA = (AGC+1)/2 form a single family of dependence measures with three exact bridges: CMA reduces to AUC for dichotomous Y (Corollary 2.3), AGC equals Spearman's rho when X and Y are continuous (Proposition 2.10), and for continuous Y, CMA = 6∫₀¹ α(1−α) AUC^(α)(X,Y) dα, where AUC^(α) is the AUC for the outcome dichotomized at the α-quantile (Theorem 2.5). The paper also establishes asymptotic normality of the sample versions and a DeLong-type test for equality of AGC/CMA values for competing predictors of a shared outcome, nesting tests for AUC and for Spearman's rho as special cases.

What carries the argument

The machinery is the mid distribution function (MDF), defined by F̄(x) = P(X<x) + ½P(X=x), which assigns tie mass evenly. The random variables F̄(X) and Ḡ(Y) are called grades, and AGC is the covariance of these grades divided by the variance of the outcome grade — equivalently, the slope of the mid-rank regression of Y on X. The granularity γ(X)=P(X=X'=X'') tracks how discrete a variable is and governs symmetry: AGC is symmetric in X and Y if and only if γ(X)=γ(Y). The asymptotic covariance formulas are built from kernels K involving the bivariate MDF and the outcome granularity.

Load-bearing premise

All inference tools in the paper flow from a central-limit theorem the authors import from an unpublished manuscript and do not prove here; if that theorem is wrong or its conditions unmet, the tests and confidence intervals are invalid.

What would settle it

Simulate n=100 independent standard normal pairs, compute AGC_n and the plug-in variance, and check whether 95% confidence intervals contain the theoretical Spearman rho value at the nominal rate; a coverage far from 95% would falsify the quoted central limit theorem. Alternatively, take Y as a discretized increasing step function of continuous X and verify numerically that AGC=1 to machine precision; any deviation refutes the perfect-predictor property.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • One estimator and one test now apply to binary, ordinal, mixed discrete-continuous, and continuous outcomes, replacing separate AUC and rank-correlation toolboxes.
  • For continuous Y, CMA is exactly 6∫₀¹ α(1−α) AUC^(α) dα, so any method that estimates AUC at a threshold can be integrated to yield a continuous-outcome measure.
  • The DeLong-type test compares competing predictors of a shared outcome without dichotomization, reducing to the classical test for binary Y.
  • AGC hits 1 exactly when Y is a monotone function of X, even with discrete Y, resolving the attainability problem that makes symmetric rank correlations misleading in discrete settings.
  • The case studies demonstrate the measures in practice: they rank LLM uncertainty metrics against correctness scores and reveal that a data-driven weather model beats a physics-based model near the equator but not toward the poles.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The AGC construction could be adapted to produce asymmetric analogues of other symmetric concordance measures, such as Blomqvist's beta, by changing the population quantity in the numerator while keeping the same grade denominator.
  • The mixture representation suggests 'partial CMA' analogs of partial AUC, defined by integrating α(1−α)AUC^(α) over a restricted quantile range to focus on a clinically or operationally relevant outcome region.
  • Because HAC-consistent variance estimates are available, the DeLong-type test could be extended to dependent data beyond the weather example, e.g., to compare predictive models on spatial or longitudinal outcomes.
  • The paper's stance that asymmetric measures are preferable when X is a predictor and Y an outcome, combined with the attainability result, implies that any strictly symmetric dependence measure will fail to reach 1 under perfect monotone dependence in discrete settings — a trade-off that should be explicit in textbook treatments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces asymmetric grade correlation (AGC) and the coefficient of monotone association (CMA), defined via mid-distribution transforms. It shows that CMA reduces to AUC when Y is dichotomous and that AGC equals Spearman's rho when X and Y are continuous, and it provides an integral representation of CMA as a weighted average of AUCs of dichotomized outcomes. Plug-in estimators are proposed, with normal limit theorems and DeLong-type tests for comparing predictors on a shared outcome. Two case studies (LLM uncertainty metrics, weather prediction) illustrate the methodology.

Significance. The population-level definitions and bridge results are clean and potentially valuable: they unify AUC and Spearman-type inference for discrete, mixed, and continuous outcomes, and the perfect-monotone-predictor property is a genuine improvement over symmetric rank correlations. The plug-in estimators are natural, computationally efficient, and the paper ships code for reproduction. However, all inferential results in Section 3 rest on an unproved, imported joint CLT (Proposition B.1), and one of the listed properties (Theorem 2.8(b)) is false as stated. If the imported result is supplied or published and the property corrected, the paper would be a substantial contribution to the dependence-measure literature.

major comments (3)
  1. [§2.2, Theorem 2.8(b)] The stated equivalence is false. From (6), AGC(X,Y) = r sqrt((1-gamma_X)/(1-gamma_Y)) and AGC(Y,X) = r sqrt((1-gamma_Y)/(1-gamma_X)), where r = cor(Fbar(X),Gbar(Y)). If r = 0 (e.g., X and Y independent), both AGCs are 0 even when gamma(X) != gamma(Y). The correct characterization is equality iff gamma(X) = gamma(Y) or AGC(X,Y) = 0. The same incorrect 'if and only if' appears in the sentence after (6). This affects the 'modulated symmetry' discussion and the proof 'Immediate from (6) and (9)' is incomplete.
  2. [Appendix B, Proposition B.1; §3.2] Proposition B.1 is stated without proof and imported from Pohle et al. (2025), an unpublished manuscript. Theorems 3.4 and 3.6, the consistency of the plug-in variance estimator Sigma_hat_n asserted after the four-step procedure in Section 3.2, and the DeLong-type tests in Section 3.3 all follow from Proposition B.1 via the delta method. Since these results generate every confidence interval and p-value in the paper, the inference is not verifiable as presented. At minimum, the authors must provide a self-contained proof of Proposition B.1 (or of the required special cases), including regularity conditions and nondegeneracy of the limiting covariance; citing an unpublished manuscript is insufficient.
  3. [§3.2 (HAC remark); §4.2] Section 3.2 states that Proposition B.1 generalizes to dependent time series and suggests using HAC estimators, and Section 4.2 uses this extension to produce p-values for the WeatherBench comparison. No mixing or moment conditions are stated, and no proof is given. As the weather case study's conclusions rest on these p-values, this extension must be either proved or clearly separated as heuristic and not load-bearing.
minor comments (5)
  1. [§3.3, chi-square paragraph] 'Theorem 2.5' should read 'Theorem 3.6' when justifying the chi-square test statistic.
  2. [§1, after definitions] The range of AGC is printed as '[−1,−1]' and should be '[−1,1]'.
  3. [§2.2, after Eq. (6)] The sentence 'AGC(X,Y) = AGC(Y,X) ... if, and only if, gamma(X) = gamma(Y)' repeats the false statement in Theorem 2.8(b); see major comment.
  4. [References] 'Pohle et al. (2025)' and 'Pohle and Wermuth (2025)' are listed as unpublished manuscripts without preprint identifiers or URLs; please provide public versions or update status.
  5. [Figure 3 caption] The phrase 'reversed orientation of RCE' could be made more explicit (e.g., 'RCE is plotted with its sign reversed so that larger values are better').

Circularity Check

0 steps flagged

No significant circularity: AGC and CMA are defined directly; their bridges to AUC and Spearman's rho are proved from the definitions, and the Section 3 CLTs rest on an external (non-overlapping-author) theorem, which is a verification gap rather than a circular reduction.

full rationale

The central measures are introduced as explicit definitions: AGC(X,Y)=cov(Fbar(X),Gbar(Y))/var(Gbar(Y)) (eq. 5) and CMA=(AGC+1)/2 (eq. 7). The headline reductions are derived, not fitted: Corollary 2.3 ('If Y is dichotomous, then CMA(X,Y)=AUC(X,Y)') is 'Immediate from (3) and (9)', and Proposition 2.10's equality AGC=rho_S in the continuous case follows from (1), (5), (6), (7). Theorem 2.5's mixture representation is proved by Fubini in Appendix A. No parameter is estimated to force these identities, and no existing measure is renamed: the paper explicitly relates AGC to the grade correlation rho_G of Pohle et al. (2025) and to Somers' D/AKC. The only load-bearing external input is the joint CLT in Proposition B.1, imported from the unpublished Pohle et al. (2025). The paper states: 'We note that the sample coefficients and 1-gamma_n are asymptotically equivalent to versions of the U-statistic in eq. (12) of Pohle et al. (2025). Therefore, Proposition A.3 and Lemma B.1 of Pohle et al. (2025) yield the following result.' This is an omitted-proof/verification concern, and the acknowledgments admit that Pohle et al. 'supplied the technical tools in the proofs of the asymptotic results in our Section 3'. However, this is not circularity under the stated criteria: Pohle et al. has no author overlap with this paper, the cited theorem is not the target AGC/CMA CLT but a U-statistic CLT for rank correlations with its own stated assumptions, and the delta method applied in Theorem 3.6 is a standard computation from that external result. Self-citations to Gneiting and Walz (2022) appear only as contextual remarks (CPA comparison, WeatherBench 1) and do not carry the derivation. Accordingly, no circular step can be exhibited; score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 2 invented entities

The central measure has no fitted parameters: AGC and CMA are defined directly from the joint distribution, and the bridges to AUC and Spearman's rho are consequences of the definitions. The main unstated inputs are the technical CLT results and the consistency of the variance estimator, both borrowed from an unpublished manuscript by Pohle and coauthors. There are no free parameters fitted to data in the derivation of the measure or its asymptotics.

axioms (3)
  • domain assumption Proposition B.1 (from Pohle et al., 2025): joint CLT for sample grade correlations and granularities under i.i.d. sampling.
    Theorem 3.4 and Theorem 3.6 in Section 3 are proved by applying the delta method to Proposition B.1. The proposition itself is not proved in the paper; it is simply stated as a result from an unpublished manuscript shared privately.
  • domain assumption Consistency of the multi-step plug-in variance estimator Σ̂_n (Pohle et al., 2025).
    Section 3.2 states 'Under Scenario 3.5 it holds that Σ̂_n → Σ in probability as n → ∞ (Pohle et al., 2025)'. No proof is given in this paper; the test statistics at (41) and (43) rely on this consistency.
  • domain assumption Dependent time-series extension of Proposition B.1 with HAC estimates (Pohle et al., 2025; Newey and West, 1987).
    Section 3.2 claims that a more general version of Proposition B.1 holds in dependent settings and that HAC estimates are consistent, citing Pohle et al. for implementation and proof. This is used in the WeatherBench precipitation case study.
invented entities (2)
  • AGC (asymmetric grade correlation) no independent evidence
    purpose: Asymmetric measure of monotone dependence of Y on X; slope of the mid-rank regression of Y on X.
    A new named measure, but it is a rescaling of the known grade correlation and rank-rank regression slope. No independent falsifiable prediction is provided outside the paper.
  • CMA (coefficient of monotone association) no independent evidence
    purpose: Affine transform of AGC (CMA = (AGC+1)/2) designed to lie in [0,1] and to reduce to AUC for binary Y and to (ρ_S+1)/2 for continuous X,Y.
    A new named measure, but again a deterministic transform of AGC rather than an independent postulate. It has no external falsifiable handle beyond the statistical properties proved in the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 3993 in / 3797 out tokens · 189524 ms · 2026-08-04T08:55:38.426966+00:00 · methodology

0 comments
read the original abstract

The assessment of monotone dependence between random variables is a classical problem in statistics and a gamut of application domains. Consequently, researchers have sought measures of association that are invariant under strictly increasing transformations of the margins, with the extant literature being splintered. For continuous variables, symmetric rank correlation coefficients, such as Spearman's Rho and Kendall's Tau, have been studied at great length in the statistical literature. For dichotomous outcomes, the asymmetric area under the curve (AUC) measure is used to assess monotone dependence. We unify and complete thus far disconnected strands of literature, by establishing common population level theory, common estimators, and common tests that bridge continuous and dichotomous settings and apply to all linearly ordered outcomes. Originating in the biomedical literature, the C index provides a bridge between AUC, to which it reduces for a dichotomous outcome, and Kendall's Tau, to which it relates linearly under continuity. To establish the same kind of bridge between AUC and Spearman's Rho, we introduce asymmetric grade correlation, AGC$(X,Y)$, as the covariance of the mid distribution function transforms, or grades, of $X$ and $Y$, divided by the variance of the grade of $Y$. The coefficient of monotone association then is CMA$(X,Y) = \frac{1}{2} ($AGC$(X,Y) + 1)$. When $X$ and $Y$ are continuous, AGC is symmetric and equals Spearman's Rho. When $Y$ is dichotomous, CMA equals AUC. We establish central limit theorems for the sample versions of these measures, and we develop tests of DeLong type for their equality with a shared outcome $Y$. In case studies, we assess progress in data-driven weather prediction and evaluate methods of uncertainty quantification for large language models.

Figures

Figures reproduced from arXiv: 2510.17994 by Andreas Eberl, Eva-Maria Walz, Tilmann Gneiting.

Figure 1
Figure 1. Figure 1: Histograms of one-sided p-values in tests of equal AGC in the first simulation example, where AGC equals Spearman’s ρS, using a) the method of Myers and Sirois (2014) with the Meng et al. (1992) adjustment for correlation, and b) our general test of DeLong type with the test statistic at (43). The sample size n equals 50, 100, 500, 1000, and 5000 (left to right), and the number of Monte Carlo replicates is… view at source ↗
Figure 2
Figure 2. Figure 2: As in Figure 1b), but showing two-sided p-values in the second simulation example. When the outcome Y is binary, the null hypothesis at (42) can be written equivalently as H0 : AUC(X(1), Y ) = AUC(X(2), Y ) in terms of the AUC measure at (3). As noted, our test and the classical test of DeLong et al. (1988) for the equality of two AUC values differ in the choice of the variance estimate only, and it is kno… view at source ↗
Figure 3
Figure 3. Figure 3: Scatterplot of CMA and RCE in the 180 experimental configurations from [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Predictive ability of WeatherBench 2 HRES, WeatherBench 2 GraphCast, and persistence forecasts of 24-hour precipitation accumulation in terms of a) RMSE skill relative to the WeatherBench 2 climatology forecast, b) ACC, c) SEEPS skill, and d) CMA skill. The performance measures are averaged over the 240 grid cells in each latitude band. to capture day-to-day variability but tends to have smaller errors tha… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of WeatherBench 2 GraphCast and HRES forecasts of 24-hour precipitation accumulation by latitude bands. The boxplots show the interquartile range (IQR) of the difference in CMA in shades of red, and the associated one-sided p-value for the test statistic at (43) in shades of blue, considering the 240 grid cells in each latitude band, respectively. Positive CMA differences and small p-values indi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Altman, D. G. and Royston, P. (2006). The cost of dichotomising continuous variables. British Medical Journal , 332:1080

    2006

  2. [2]

    and Roudaki, P

    Azadkia, M. and Roudaki, P. (2025). A new measure of dependence: I ntegrated R ^2 . Preprint, https://arxiv.org/abs/2505.18146

    arXiv 2025

  3. [3]

    Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology , 12:387--415

    1975

  4. [4]

    C., Magnusson, L., Gascon, E., Maier-Gerber, M., Janou s ek, M., Rodwell, M., Pinault, F., Dramsch, J

    Ben Bouall\` e gue, Z., Clare, M. C., Magnusson, L., Gascon, E., Maier-Gerber, M., Janou s ek, M., Rodwell, M., Pinault, F., Dramsch, J. S., Lang, S. T., Raoult, B., Rabier, F., Chevallier, M., Sandu, I., Dueben, P., Chantry, M., and Pappenberger, F. (2024). The rise of data-driven weather forecasting: A first statistical assessment of machine learning-ba...

    2024

  5. [5]

    and Dassios, A

    Bergsma, W. and Dassios, A. (2014). A consistent test of independence based on a sign covariance related to K endall’s tau. Bernoulli , 20:1006--1028

    2014

  6. [6]

    and Roberts, D

    Best, D. and Roberts, D. (1975). Algorithm AS 89: T he upper tail probabilities of S pearman's rho. Journal of the Royal Statistical Society Series C (Applied Statistics) , 24:377--379

    1975

  7. [7]

    Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., and Tian, Q. (2023). Accurate medium-range global weather forecasting with 3d neural networks. Nature , 619:533--538

    2023

  8. [8]

    Borkowf, C. B. (2002). Computing the nonnull asymptotic variance and the asymptotic relative efficiency of S pearman's rank correlation. Computational Statistics & Data Analysis , 39:271--286

    2002

  9. [9]

    Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition , 30:1145--1159

    1997

  10. [10]

    Chatterjee, S. (2021). A new coefficient of correlation. Journal of the American Statistical Association , 116:2009--2022

    2021

  11. [11]

    and Wilhelm, D

    Chetverikov, D. and Wilhelm, D. (2023). Inference for rank-rank regressions. Preprint, https://arxiv.org/abs/2310.15512

    Pith/arXiv arXiv 2023

  12. [12]

    R., DeLong, D

    DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics , 44:837--845

    1988

  13. [13]

    V., Pencina, M

    Demler, O. V., Pencina, M. J., Cook, N. R., and D'Agostino Sr, R. B. (2017). Asymptotic distribution of AUC , NRI s, and IDI based on theory of U -statistics. Statistics in Medicine , 36:3334--3360

    2017

  14. [14]

    Dem s ar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research , 7:1--30

    2006

  15. [15]

    and Musch, J

    Diedenhofen, B. and Musch, J. (2015). cocor: A comprehensive solution for the statistical comparison of correlations. PLoS ONE , 10:e0121945

    2015

  16. [16]

    Embrechts, P., McNeil, A., and Straumann, D. (2002). Correlation and dependence in risk management: P roperties and pitfalls. In Dempster, M. A. H., editor, Risk Management: Value at Risk and Beyond , pages 176--223. Cambridge University Press

    2002

  17. [17]

    Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters , 27:861--874

    2006

  18. [18]

    Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika , 10:507--521

    1915

  19. [19]

    and Pohle, M.-O

    Fissler, T. and Pohle, M.-O. (2023). Generalised covariances and correlations. Preprint, https://arxiv.org/abs/2307.03594

    Pith/arXiv arXiv 2023

  20. [20]

    and Schmid, F

    Gai er, S. and Schmid, F. (2010). On testing equality of pairwise rank correlations in a multivariate random vector. Journal of Multivariate Analysis , 101:2598--2615

    2010

  21. [21]

    and Lafaye de Micheaux, P

    Geenens, G. and Lafaye de Micheaux, P. (2022). The H ellinger correlation. Journal of the American Statistical Association , 117:639--653

    2022

  22. [22]

    G., and R \'e millard, B

    Genest, C., Ne s lehov \'a , J. G., and R \'e millard, B. (2013). On the estimation of S pearman’s rho and related tests of independence for possibly discontinuous multivariate data. Journal of Multivariate Analysis , 117:214--228

    2013

  23. [23]

    and Raffinetti, E

    Giudici, P. and Raffinetti, E. (2025). RGA : A unified measure of predictive accuracy. Advances in Data Analysis and Classification , 19:67--93

    2025

  24. [24]

    Probabilistic measures afford fair comparisons of AIWP and NWP model output

    Gneiting, T., Biegert, T., Kraus, K., Walz, E.-M., Jordan, A. I., and Lerch, S. (2025). Probabilistic measures afford fair comparisons of AIWP and NWP model output. Preprint, https://doi.org/10.48550/arXiv.2506.03744

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.03744 2025

  25. [25]

    and Vogel, P

    Gneiting, T. and Vogel, P. (2022). Receiver operating characteristic ( ROC ) curves: E quivalences, beta model, and minimum distance estimation. Machine Learning , 111:2147--2159

    2022

  26. [26]

    and Walz, E.-M

    Gneiting, T. and Walz, E.-M. (2022). Receiver operating characteristic ( ROC ) movies, universal ROC ( UROC ) curves, and coefficient of predictive ability ( CPA ). Machine Learning , 111:2769--2797

    2022

  27. [27]

    E., Lee, K

    Harrell Jr, F. E., Lee, K. L., and Mark, D. B. (1996). Multivariable prognostic models: I ssues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine , 15:361--387

    1996

  28. [28]

    D., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R

    Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Hor \'a nyi, A., Mu \ n oz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., Chiara, G. D., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuent...

    2020

  29. [29]

    Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution. Annals of Mathematical Statistics , 19:293--325

    1948

  30. [30]

    Huang, X., Li, S., Yu, M., Sesia, M., Hassani, H., Lee, I., Bastani, O., and Dobriban, E. (2024). Uncertainty in language models: A ssessment through rank-calibration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics

    2024

  31. [31]

    Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (2017). TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers . Association for Computational Linguistics

    2017

  32. [32]

    Kendall, M. G. (1938). A new measure of rank correlation. Biometrika , 30:81--93

    1938

  33. [33]

    and Sampson, A

    Kimeldorf, G. and Sampson, A. R. (1978). Monotone dependence. Annals of Statistics , 6:895--903

    1978

  34. [34]

    Kruskal, W. H. (1958). Ordinal measures of association. Journal of the American Statistical Association , 53:814--861

    1958

  35. [35]

    Kuhn, L., Gal, Y., and Farquhar, S. (2023). Semantic uncertainty: L inguistic invariances for uncertainty estimation in natural language generation. In Eleventh International Conference on Learning Representations

    2023

  36. [36]

    K \"u nsch, H. R. (1989). The jackknife and the bootstrap for general stationary observations. Annals of Statistics , 17:1217--1241

    1989

  37. [37]

    Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., Merose, A., Hoyer, S., Holland, G., Vinyals, O., Stott, J., Pritzel, A., Mohamed, S., and Battaglia, P. (2023). Learning skillful medium-range global weather forecasting. Science , 382:1416--1421

    2023

  38. [38]

    Lancaster, H. (1963). Correlation and complete dependence of random variables. Annals of Mathematical Statistics , 34:1315--1321

    1963

  39. [39]

    A., Simmons, A., Vamborg, F., and Rodwell, M

    Lavers, D. A., Simmons, A., Vamborg, F., and Rodwell, M. J. (2022). An evaluation of ERA5 precipitation for climate monitoring. Quarterly Journal of the Royal Meteorological Society , 148:3152--3165

    2022

  40. [40]

    Lee, K., Chang, M.-W., and Toutanova, K. (2019). Latent retrieval for weakly supervised open domain question answering. In Korhonen, A., Traum, D., and M \`a rquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6086--6096

    2019

  41. [41]

    Lehmann, E. L. and Romano, J. P. (2022). Testing Statistical Hypotheses . Springer, 4th edition

    2022

  42. [42]

    Lin, Z., Trivedi, S., and Sun, J. (2024). Generating with confidence: U ncertainty quantification for black-box large language models. Transactions on Machine Learning Research

    2024

  43. [43]

    Meng, X.-L., Rosenthal, R., and Rubin, D. B. (1992). Comparing correlated correlation coefficients. Psychological Bulletin , 111:172--175

    1992

  44. [44]

    and Sirois, M

    Myers, L. and Sirois, M. J. (2014). Spearman correlation coefficients, differences between. In Wiley Stats\-Ref: Statistics Reference Online . Wiley Online Library, https://doi.org/10.1002/9781118445112.stat02802

    doi:10.1002/9781118445112.stat02802 2014

  45. [45]

    Ne s lehov \'a , J. (2007). On rank correlation measures for non-continuous random variables. Journal of Multivariate Analysis , 98:544--567

    2007

  46. [46]

    and West, K

    Newey, W. and West, K. (1987). A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica , 55:703--708

    1987

  47. [47]

    Newson, R. (2002). Parameters behind ``nonparametric'' statistics: K endall's tau, S omers’ D and median differences. The Stata Journal , 2(1):45--64

    2002

  48. [48]

    and Kowalczyk, T

    Niewiadomska-Bugaj, M. and Kowalczyk, T. (2005). On grade transformation and its implications for copulas. Brazilian Journal of Probability and Statistics , 19:125--137

    2005

  49. [49]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2021). Training language models to follow instructions with human feedback. In 35th Conference on Neural Inf...

    2021

  50. [50]

    Pencina, M. J. and D'Agostino, R. B. (2004). Overall C as a measure of discrimination in survival analysis: M odel specific population value and confidence interval estimation. Statistics in Medicine , 23:2109--2123

    2004

  51. [51]

    and Wermuth, J.-L

    Pohle, M.-O. and Wermuth, J.-L. (2025). Proper correlation coefficients for discrete random variables. Unpublished manuscript

    2025

  52. [52]

    Pohle, M.-O., Wermuth, J.-L., and Wei , C. H. (2025). Statistical inference for rank correlations. Unpublished manuscript

    2025

  53. [53]

    Python language reference

    Python Software Foundation (2025). Python language reference. Available at http://www.python.org

    2025

  54. [54]

    R : A language and environment for statistical computing

    R Core Team (2025). R : A language and environment for statistical computing. Available at http://www.r-project.org

    2025

  55. [55]

    T., Ebert-Uphoff, I., and Stewart, J

    Radford, J. T., Ebert-Uphoff, I., and Stewart, J. Q. (2025). A comparison of AI weather prediction and numerical weather prediction models for 1--7-day precipitation forecasts. Weather and Forecasting , 40:561--575

    2025

  56. [56]

    Raffinetti, E. (2023). A rank graduation accuracy measure to mitigate artificial intelligence risks. Quality & Quantity: International Journal of Methodology , 57:131--150

    2023

  57. [57]

    Rainio, O., Teuho, J., and Kl \'e n, R. (2024). Evaluation metrics and statistical tests for machine learning. Scientific Reports , 14:6086

    2024

  58. [58]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. Preprint, https://doi.org/10.48550/arXiv.1606.05250

    Pith/arXiv arXiv doi:10.48550/arxiv.1606.05250 2016

  59. [59]

    D., Scher, S., Weyn, J

    Rasp, S., Dueben, P. D., Scher, S., Weyn, J. A., Mouatadid, S., and Thuerey, N. (2020). Weather B ench: A benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems , 12:e2020MS002203

    2020

  60. [60]

    Rasp, S., Hoyer, S., Merose, A., Langmore, I., Battaglia, P., Russell, T., Sanchez-Gonzalez, A., Yang, V., Carver, R., Agrawal, S., Chantry, M., Ben Bouall\` e gue, Z., Dueben, P., Bromberg, C., Sisk, J., Barrington, L., Bell, A., and Sha, F. (2024). Weather B ench 2: A benchmark for the next generation of data-driven global weather models. Journal of Adv...

    2024

  61. [61]

    R \'e nyi, A. (1959). On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica , 10:441--451

    1959

  62. [62]

    N., Reshef, Y

    Reshef, D. N., Reshef, Y. A., Finucane, H. K., Grossman, S. R., McVean, G., Turnbaugh, P. J., Lander, E. S., Mitzenmacher, M., and Sabeti, P. C. (2011). Detecting novel associations in large data sets. Science , 334:1518--1524

    2011

  63. [63]

    Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., and M \"u ller, M. (2011). p ROC : A n open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics , 12:1--8

    2011

  64. [64]

    J., Richardson, D

    Rodwell, M. J., Richardson, D. S., Hewson, T. D., and Haiden, T. (2010). A new equitable score suitable for verifying precipitation in numerical weather prediction. Quarterly Journal of the Royal Meteorological Society , 136:1344--1363

    2010

  65. [65]

    Scarsini, M. (1984). On measures of concordance. Stochastica , 8:201--218

    1984

  66. [66]

    and Wolff, E

    Schweizer, B. and Wolff, E. F. (1981). On nonparametric measures of dependence for random variables. Annals of Statistics , 9:879--885

    1981

  67. [67]

    Z., and Majumdar, A

    Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., and Majumdar, A. (2025). A survey on uncertainty quantification of large language models: T axonomy, open research challenges, and future directions. ACM Computing Surveys , 58(3):63

    2025

  68. [68]

    Somers, R. H. (1962). A new asymmetric measure of association for ordinal variables. American Sociological Review , 27:799--811

    1962

  69. [69]

    Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology , 15:72--101

    1904

  70. [70]

    and Xu, W

    Sun, X. and Xu, W. (2014). Fast implementation of DeLong ’s algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters , 21:1389--1393

    2014

  71. [71]

    Swets, J. A. (1988). Measuring the accuracy of diagnostic systems. Science , 240:1285--1293

    1988

  72. [72]

    J., Rizzo, M

    Sz \'e kely, G. J., Rizzo, M. L., and Bakirov, N. K. (2007). Measuring and testing dependence by correlation of distances. Annals of Statistics , 35:2769--2794

    2007

  73. [73]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa...

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.09288 2023

  74. [74]

    Weihs, L., Drton, M., and Meinshausen, N. (2018). Symmetric rank covariances: A generalized framework for nonparametric measures of dependence. Biometrika , 105:547--562

    2018

  75. [75]

    Woodbury, M. A. (1940). Rank correlation when there are equal variates. Annals of Mathematical Statistics , 11:358--362

    1940

  76. [76]

    Xu, J. (2024). On the bias in the AUC variance estimate. Pattern Recognition Letters , 178:62--68

    2024