REVIEW 4 major objections 5 minor 49 references
This paper claims that the correct correlation method for a variable pair can be chosen automatically by variable type, and that doing so changes substantive results on real data.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 05:13 UTC pith:CROQGK2D
load-bearing objection Useful, well-engineered tools paper with a real empirical mini-study, but the simulations validate estimators, not the auto-pipeline, and low-cardinality counts get mislabeled as ordinal. the 4 major comments →
smartcor: Intelligent Correlation Method Selection for Mixed Variable Types
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the 'one correlation fits all' default of statistical software is wrong for most real variable pairs, and that a small, transparent decision rule can fix it. The paper establishes two algebraic identities—Pearson's r equals the point-biserial coefficient for continuous-binary pairs and the phi coefficient for binary-binary pairs under 0/1 coding—and then shows that for the remaining seven pairings, the choice of estimator matters: Pearson attenuates ordinal associations by roughly 2–21% (up to ~37% for dichotomized data), and Cramér's V or related measures are required for nominal variables because Pearson is undefined there. The implemented tool routes each pair to
What carries the argument
The load-bearing object is a decision table: the five variable types (continuous, binary, ordinal, count, categorical) generate ten unordered pairs, and each pair maps to a recommended estimator among 14 supported methods. Two formal equivalences carry the 'Pearson is fine' part: Theorem 2.1 shows Pearson equals point-biserial for continuous-binary, Theorem 2.2 shows Pearson equals phi for binary-binary. For ordinal-involving pairs, a likelihood-ratio goodness-of-fit test of the bivariate-normal threshold model chooses between polychoric/polyserial (if normality is not rejected) and Kendall/Spearman (if it is). The package also separates signed correlation from unsigned association, applying
Load-bearing premise
The default 'auto' choice between latent-variable and rank-based correlation relies on a likelihood-ratio test whose failure to reject normality is treated as evidence for normality, even though at small samples (n around 30) that test has low power, so non-normal latent distributions will often be mistakenly accepted and a biased estimator selected.
What would settle it
Simulate, say, 1,000 datasets of n=32 from a bivariate distribution with a strongly skewed latent (e.g., exponential margins transformed to a normal copula with rho=0.5) and ordinal variables with 5 categories each; if smart_cor's 'auto' option selects polychoric in the majority of cases and the polychoric estimate has bias >0.1 while Kendall's tau is unbiased, the automatic selection logic is not doing what it claims for small samples.
If this is right
- If the paper is right, default Pearson correlation tables in software are misleading for the majority of mixed-type pairs; correcting the default would change reported effect sizes and significance in a substantial minority of published correlations.
- The algebraic equivalences mean practitioners gain nothing by switching methods for continuous-binary and binary-binary pairs with 0/1 coding; the gain is concentrated in ordinal and categorical pairs.
- Type-aware selection makes it practical to compute a single mixed-type correlation matrix with nine or more distinct methods without manual specification.
- The data-driven latent-normality test means the choice between latent-variable and rank-based estimators can be audited and overridden, reducing reliance on untested assumptions.
Where Pith is reading between the lines
- The auto latent-normality rule inherits the LR test's low power at small n; a natural extension is a sample-size-aware fallback (e.g., require a minimum n or a confidence margin before selecting polychoric). This is an editorial inference, not a paper claim.
- Because type detection is heuristic (e.g., integer-valued 'count' vs 'continuous'), the package's results can depend on the ordinal_threshold; a testable extension is a sensitivity analysis that varies the threshold and reports how many cells change method.
- The correlation-versus-association distinction suggests that presenting signed and unsigned values in one matrix could mislead readers who rank them; a stronger design would separate the two blocks in output. This is a design suggestion beyond the paper's stated scope.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces smartcor (R) and pysmartcor (Python), a package that automatically detects whether each variable is continuous, binary, ordinal, count, or categorical, and then selects one of 14 correlation or association methods for each pair. The main contributions claimed are: (i) algebraic proofs that Pearson correlation equals the point-biserial coefficient for continuous–binary pairs and the phi coefficient for binary–binary pairs; (ii) a simulation study comparing estimator bias across variable-type pairs and latent distributions; (iii) a data-driven latent-normality test that chooses between polychoric/polyserial and rank-based methods; (iv) a mini-study of economics papers showing that 92.3% of correlation calls are Pearson; and (v) case studies on mtcars and General Social Survey data illustrating substantive differences between naive and type-aware analysis.
Significance. If the central claim is fully supported, the package would fill a real practical gap: applied researchers routinely compute Pearson correlations on mixed-type data without checking measurement levels. The paper's strengths include clean algebraic proofs (Appendix A), a transparent and explainable decision architecture with user override, and simulations that separate skew direction from symmetric heavy-tailed departures. The R and Python implementations are distributed through standard repositories, and the tidy API is a genuine usability contribution. However, the paper currently claims more than it validates: the Monte Carlo simulations fix variable types and do not exercise the automatic type-detection and selection pipeline end-to-end, and one concrete detection-rule bug (low-cardinality counts) affects the paper's own flagship example. These issues are fixable, but they currently limit the strength of the paper's main claim.
major comments (4)
- [§3.2 (rules 5–6), §1, §3.5] The type-detection order is internally inconsistent. Rule 5 labels any numeric vector with ≤ ordinal_threshold unique values as ordinal before Rule 6 can label non-negative integer-valued vectors as count. Thus a count variable with ≤10 distinct values — e.g., number of children, previous jobs, or carburetors — is labeled ordinal. This contradicts §1's 'all count variables are treated as continuous variables' and §3.2's claim that the count label 'never changes the selected method or the estimate.' It does change the method: mtcars$carb has 6 unique values and is labeled ordinal in §3.5; the reported carb–gear value is 0.098 (Kendall) whereas treating carb as continuous would give Pearson ≈ 0.27. The paper attributes this gap to ordinal structure, but part of it is an artifact of the detection heuristic. Since the Section 2 simulations never call detect_type(), the auto-detection step it
- [§2.5, §3.4, §3.6] The default assume_latent_normal='auto' uses the Jöreskog LR goodness-of-fit test and selects polychoric/polyserial when the test fails to reject latent normality. At small n, this test has low power, so non-normal latent distributions will often fail to reject and the auto mode will choose the latent-variable estimator despite the mismatch. The paper explicitly acknowledges this in §2.5 ('Failure to reject is of course not equivalent to accepting normality, particularly at small sample sizes'), but the default behavior does not reflect the caveat. The mtcars example (§3.6) illustrates the risk: at n=32, compare_methods() recommends polychoric for gear–carb on the basis of the LR test. Either the default should be more conservative at small n, the output should warn about low power, or the paper should provide a power analysis justifying the current threshold.
- [Abstract; §2.3–§2.4] The abstract states that 'Monte Carlo simulations validate the selection logic,' but the simulations in Section 2 fix the variable types and compare estimators; they do not feed raw data through detect_type() and the full selection rules. The end-to-end pipeline has three components — type detection, method selection, and the latent-normality decision — and none of these is evaluated as a system. For example, there is no measurement of detect_type() misclassification rates, no sensitivity analysis for ordinal_threshold, and no evaluation of the auto latent-normality decision's operating characteristics. The claim should be softened or an end-to-end simulation should be added.
- [§2 (paragraph beginning 'Some binary variables may be categorical')] The text states: 'when a scalar variable with two (and only two) unique values exists, it is a binary categorical variable and must use an association measure instead of correlation measures.' Taken literally, this contradicts Table 1 and the package's own behavior: Table 1 lists point-biserial, phi, tetrachoric, and Yule's Q for binary variables, and the GSS case study (§4.3) computes point-biserial correlation for sex. If the sentence were implemented, no binary variable could ever receive a correlation coefficient. The intended distinction is presumably between binary variables with a meaningful 0/1 ordering and binary variables whose two labels are nominal categories, but the sentence as written is internally inconsistent and should be rewritten.
minor comments (5)
- [Table 1 vs. Table 2] Table 1 lists Spearman and Kendall as available methods for continuous–binary pairs, but Table 2 gives no alternative for continuous–binary. Clarify whether rank methods are appropriate for this pairing and, if so, list them in Table 2.
- [§2.3 (Figures 2 and 4)] Spearman's deviation from the latent ρ is labeled 'attenuation bias,' but Spearman estimates a rank-based monotonic association, not the linear correlation. The text acknowledges this scale issue for Kendall but not for Spearman. Consider relabeling the figures or adding an explicit caveat.
- [§3.2] The ordinal_threshold default of 10 is presented without any sensitivity analysis. Because the threshold directly controls the low-cardinality-count misclassification discussed above, a short sensitivity check (e.g., threshold 5, 10, 20) would help users understand the heuristic's robustness.
- [§4.2] The text says the surviving Pearson correlations 'underestimate associations involving binary variables (Section 2.2),' but Section 2.2 proves Pearson equals the point-biserial coefficient for continuous–binary data. The underestimation issue arises specifically when binary variables are treated as dichotomized continuous variables (tetrachoric case), not generally. The wording should be corrected.
- [§3.5] In the mtcars example, hp is detected as count and then 'counts are treated as numeric continuous variables' for method selection. This is fine, but the paper's repeated statement that the count label 'never changes the selected method' is confusing given the low-cardinality count problem. Consider restricting that claim to count variables that pass the ordinal_threshold.
Circularity Check
No circularity found: core equivalences are proved algebraically and simulations use external generative benchmarks; the count/ordinal heuristic and LR-test power are correctness/validation limitations, not circular reductions.
full rationale
The paper's central derivation is self-contained. Theorem 2.1 and Theorem 2.2 are algebraic identities proved in Appendix A: Pearson correlation on 0/1-coded data equals point-biserial and phi coefficients, with no statistical assumptions or fitted parameters. The simulations in Sections 2.3–2.4 evaluate estimator bias under explicit generative models (e.g., bivariate normal latents, skewed and heavy-tailed latents) and are externally checkable; they do not use smartcor's own outputs as inputs. The latent-normality decision relies on the Jöreskog (2005) likelihood-ratio test, an external method, and the paper explicitly acknowledges the small-sample limitation: 'Failure to reject is of course not equivalent to accepting normality, particularly at small sample sizes.' That is a validation/power caveat, not a circular step. The type-detection heuristic can label low-cardinality count variables as ordinal (e.g., mtcars$carb), which creates an internal inconsistency with the statement that all count variables are treated as continuous; however, this is a correctness/accuracy limitation rather than a circularity, because the method-selection rules are evaluated on independent real datasets and the special-purpose estimators are standard, externally validated methods rather than quantities fitted to the paper's own target result. There is no load-bearing self-citation, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The paper's overbroad claim that Monte Carlo simulations 'validate the selection logic' is incomplete since detect_type and the auto LR-test decision are not simulated end-to-end, but this under-specification is a validation gap, not a circular derivation. Score 0 reflects the absence of circularity in the core derivation.
Axiom & Free-Parameter Ledger
free parameters (3)
- ordinal_threshold =
10
- continuous-to-categorical bin count for Cramér's V =
5 (quintiles)
- alpha for latent normality test =
0.05
axioms (3)
- domain assumption Latent bivariate normality is required for polychoric/polyserial/tetrachoric to estimate the true latent correlation.
- domain assumption The LR chi-square test of bivariate normality (Jöreskog, 2005) has correct size/power for the package's auto mode.
- domain assumption Pearson's r, Spearman, Kendall, etc. are the correct measures for their respective type pairs.
read the original abstract
Pearson correlation is the default measure of association in most statistical software, yet it is only appropriate for pairs of continuous variables with a linear relationship. When variables are binary, ordinal, or categorical, specialized methods (e.g., point-biserial, polychoric, tetrachoric, and Cram\'{e}r's~$V$) may be more appropriate, but practitioners rarely know which to select. The \textbf{smartcor} package for \textbf{R} (and its companion \textbf{pysmartcor} package for \textbf{Python}) automatically detects variable types, selects the statistically appropriate correlation method for each pair, and explains its reasoning. The package supports 14~correlation and association methods covering all 10~variable-type pair combinations, and distinguishes true correlation (for ordinal and continuous pairs) from statistical association (for nominal categorical pairs). Monte~Carlo simulations validate the selection logic, and a case study with General Social Survey data demonstrates substantive differences between naive and type-aware correlation analysis.
Figures
Reference graph
Works this paper leans on
-
[1]
Benesty, J
J. Benesty, J. Chen, Y. Huang, and I. Cohen. Pearson correlation coefficient. In Noise Reduction in Speech Processing, pages 1--4. Springer, 2009
2009
-
[2]
D. G. Bonett and T. A. Wright. Sample size requirements for estimating pearson, kendall and spearman correlations. Psychometrika, 65 0 (1): 0 23--28, 2000
2000
-
[3]
M. B. Brown and J. K. Benedetti. Sampling behavior of tests for correlation in two-way contingency tables. Journal of the American Statistical Association, 72 0 (358): 0 309--315, 1977
1977
-
[4]
Casella and R
G. Casella and R. L. Berger. Statistical Inference. Duxbury, Pacific Grove, CA, 2nd edition, 2002
2002
-
[5]
N. Cliff. Ordinal Methods for Behavioral Data Analysis. Lawrence Erlbaum Associates, Mahwah, NJ, 1996
1996
-
[6]
Cram \'e r
H. Cram \'e r. Mathematical Methods of Statistics. Princeton University Press, Princeton, 1946
1946
-
[7]
E. E. Cureton. Rank-biserial correlation. Psychometrika, 21 0 (3): 0 287--290, 1956. doi:10.1007/BF02289138
-
[8]
E. C. Davenport and N. A. El-Sanhurry. Phi/phimax: Review and synthesis. Educational and Psychological Measurement, 51 0 (4): 0 821--828, 1991. doi:10.1177/001316449105100403
-
[9]
J. C. De Winter, S. D. Gosling, and J. Potter. Comparing the P earson and S pearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological Methods, 21 0 (3): 0 273--290, 2016
2016
-
[10]
F. Drasgow. Polychoric and polyserial correlations. The Encyclopedia of Statistics, 7: 0 68--74, 1986
1986
-
[11]
E. C. Fieller, H. O. Hartley, and E. S. Pearson. Tests for rank correlation coefficients. I . Biometrika, 44 0 (3/4): 0 470--481, 1957
1957
-
[12]
D. B. Flora and P. J. Curran. An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9 0 (4): 0 466--491, 2004
2004
-
[13]
J. Fox. polycor : Polychoric and Polyserial Correlations , 2022. URL https://CRAN.R-project.org/package=polycor. R package version 0.8-1
2022
-
[14]
L. A. Goodman and W. H. Kruskal. Measures of association for cross classifications. Journal of the American Statistical Association, 49 0 (268): 0 732--764, 1954
1954
-
[15]
C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, et al. Array programming with NumPy . Nature, 585 0 (7825): 0 357--362, 2020. doi:10.1038/s41586-020-2649-2
-
[16]
K. Healy. gssr : General Social Survey Data for R , 2023. URL https://kjhealy.github.io/gssr/. R package version 0.4.0
2023
-
[17]
S. L. Hershberger. Tetrachoric correlation. Encyclopedia of statistics in behavioral science, 2005
2005
-
[18]
J. D. Hunter. Matplotlib : A 2D graphics environment. Computing in Science & Engineering, 9 0 (3): 0 90--95, 2007. doi:10.1109/MCSE.2007.55
-
[19]
Iannone, J
R. Iannone, J. Cheng, B. Schloerke, E. Hughes, A. Lauer, and J. Seo. gt: Easily Create Presentation-Ready Display Tables, 2024. URL https://CRAN.R-project.org/package=gt. R package version 0.10
2024
-
[20]
K. G. J \"o reskog. On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59 0 (3): 0 381--389, 1994
1994
-
[21]
K. G. J \"o reskog. Structural equation modeling with ordinal variables using LISREL . Technical Report, 2005. URL https://www.ssicentral.com/lisrel/techdocs/ordinal.pdf. Scientific Software International, Lincolnwood, IL
2005
-
[22]
M. G. Kendall. A new measure of rank correlation. Biometrika, 30 0 (1-2): 0 81--93, 1938
1938
-
[23]
M. G. Kendall. The treatment of ties in ranking problems. Biometrika, 33 0 (3): 0 239--251, 1945
1945
-
[24]
Kiwanuka, J
F. Kiwanuka, J. Kopra, N. Sak-Dankosky, R. C. Nanyonga, and T. Kvist. Polychoric correlation with ordinal data in nursing research. Nursing Research, 71 0 (6): 0 469--476, 2022
2022
-
[25]
S. Kranz. Find economic articles with data. Database and search application, Ulm University, 2023. URL https://ejd.econ.mathematik.uni-ulm.de. Snapshot of 2024-05-30
2023
-
[26]
D. Makowski, M. S. Ben-Shachar, I. Patil, and D. L \"u decke. Methods and algorithms for correlation analysis in R . Journal of Open Source Software, 5 0 (51): 0 2306, 2020. doi:10.21105/joss.02306
-
[27]
W. McKinney. Data structures for statistical computing in Python . In Proceedings of the 9th Python in Science Conference, pages 56--61, 2010. doi:10.25080/Majora-92bf1922-00a
-
[28]
Muth \'e n and D
B. Muth \'e n and D. Kaplan. A comparison of some methodologies for the factor analysis of non-normal L ikert variables. British Journal of Mathematical and Statistical Psychology, 38 0 (2): 0 171--189, 1985
1985
-
[29]
Muth \'e n and D
B. Muth \'e n and D. Kaplan. A comparison of some methodologies for the factor analysis of non-normal L ikert variables: A note on the size of the model. British Journal of Mathematical and Statistical Psychology, 45 0 (1): 0 19--30, 1992
1992
-
[30]
U. Olsson. Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44 0 (4): 0 443--460, 1979
1979
-
[31]
U. Olsson, F. Drasgow, and N. J. Dorans. The polyserial correlation coefficient. Psychometrika, 47 0 (3): 0 337--347, 1982. doi:10.1007/BF02294164
-
[32]
K. Pearson. Notes on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58: 0 240--242, 1895
-
[33]
K. Pearson. Mathematical Contributions to the Theory of Evolution. XIII . On the Theory of Contingency and Its Relation to Association and Normal Correlation . Number 1 in Drapers' Company Research Memoirs, Biometric Series. Dulau and Co., London, 1904
1904
-
[34]
Poon and S.-Y
W.-Y. Poon and S.-Y. Lee. Maximum likelihood estimation of multivariate polyserial and polychoric correlation coefficients. Psychometrika, 52 0 (3): 0 409--430, 1987
1987
-
[35]
R : A Language and Environment for Statistical Computing
R Core Team . R : A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria, 2025. URL https://www.R-project.org/
2025
-
[36]
W. Revelle. psych : Procedures for Psychological, Psychometric, and Personality Research . Northwestern University, Evanston, IL, 2024. URL https://CRAN.R-project.org/package=psych. R package version 2.4.3
2024
-
[37]
Robinson, A
D. Robinson, A. Hayes, and S. Couch. broom : Convert Statistical Analysis Objects into Tidy Tibbles , 2024. URL https://CRAN.R-project.org/package=broom. R package version 1.0.7
2024
-
[38]
T. W. Smith, M. Davern, J. Freese, and S. Morgan. General social surveys, 1972--2024. NORC at the University of Chicago, 2024. URL https://gss.norc.org/. Machine-readable data file
1972
-
[39]
Spearman
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15 0 (1): 0 72--101, 1904
1904
-
[40]
H. Theil. Economics and information theory. 1967
1967
-
[41]
A. A. Tschuprow. Principles of the Mathematical Theory of Correlation. W. Hodge & Co., London, 1939. Translated by M. Kantorowitsch
1939
-
[42]
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al. SciPy 1.0: Fundamental algorithms for scientific computing in Python . Nature Methods, 17: 0 261--272, 2020. doi:10.1038/s41592-019-0686-2
-
[43]
M. L. Waskom. seaborn : Statistical data visualization. Journal of Open Source Software, 6 0 (60): 0 3021, 2021. doi:10.21105/joss.03021
-
[44]
H. Wickham. ggplot2 : Elegant Graphics for Data Analysis . Springer-Verlag, New York, 2016. URL https://ggplot2.tidyverse.org
2016
-
[45]
Wickham and L
H. Wickham and L. Henry. purrr: Functional Programming Tools, 2023. URL https://CRAN.R-project.org/package=purrr. R package version 1.0.2
2023
-
[46]
H. Wickham, M. Averick, J. Bryan, W. Chang, L. D. McGowan, R. Fran c ois, G. Grolemund, A. Hayes, L. Henry, J. Hester, M. Kuhn, T. L. Pedersen, E. Miller, S. M. Bache, K. M \"u ller, J. Ooms, D. Robinson, D. P. Seidel, V. Spinu, K. Takahashi, D. Vaughan, C. Wilke, K. Woo, and H. Yutani. Welcome to the tidyverse . Journal of Open Source Software, 4 0 (43):...
-
[47]
Wickham, R
H. Wickham, R. Fran c ois, L. Henry, K. M \"u ller, and D. Vaughan. dplyr: A Grammar of Data Manipulation, 2023. URL https://CRAN.R-project.org/package=dplyr. R package version 1.1.4
2023
-
[48]
G. U. Yule. On the methods of measuring association between two attributes. Journal of the Royal Statistical Society, 75 0 (6): 0 579--652, 1912
1912
-
[49]
H. Zhu. kableExtra: Construct Complex Table with kable and Pipe Syntax, 2024. URL https://CRAN.R-project.org/package=kableExtra. R package version 1.4.0
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.