REVIEW 3 major objections 6 minor 1 cited by
Permutation-Based Rank Test in the Presence of Discretization and Application in Causal Discovery with Mixed Data
T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims that permutation-based null-distribution estimation makes rank tests of cross-covariance matrices valid when some or all variables are discretized.
desk verdict A genuinely new permutation-based rank test for discretized mixed data with strong empirical support, but the core proof of the permuted-statistic consistency has a gap that leaves the theoretical Type I error claim unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the exchangeability of the trailing canonical variables under the null, combined with Equation (8), which expresses the permuted CCA scores as singular values of a matrix whose cross term $A^T (D_X^T P^T D_Y/(N-1)) B$ can be estimated from discretized data. The pairwise pseudo-likelihood (a tractable sum of pair-wise log-likelihoods, with polyserial terms for continuous-discrete pairs and polychoric terms for discrete-discrete pairs) supplies the cross-covariance estimate under a unit-variance Gaussian assumption, and a spherical parameterization of correlation matrices keeps the estimate in the pseudo-correlation space needed for SVD. This lets the permutation distribution be computed without ever applying a linear transformation to ordinal columns.
What would settle it
Generate independent standard Gaussian $X$ and $Y$, discretize both into three ordinal categories, fix a random permutation $P$, and compare the likelihood-based estimate of $(D_X^T P^T D_Y)/(N-1)$ with the actual sample value as $N$ grows; a persistent bias would falsify Theorem 5. A complementary check is that, under a true null, MPRT p-values must be uniform across repeated trials, so a Type I error rate systematically above the nominal level would refute the central claim.
Extended reading notes
Core claim
Under the null hypothesis $H_0^k$ that $\operatorname{rank}(\Sigma_{X,Y}) \le k$, the paper establishes asymptotic exchangeability of the trailing canonical variables $C_{X,k:}$ and $C_{Y,k:}$ even when $X$ or $Y$ contains discretized components. Since ordinal columns cannot be linearly transformed into canonical scores, it proves (Theorem 5) that the permuted CCA scores can instead be consistently estimated from permuted discretized data through likelihood-based cross-covariance estimation. The consequence is that the empirical permutation distribution of Anderson's likelihood-ratio statistic is an asymptotically valid null distribution, so MPRT controls Type I error in fully continuous, partially discretized, and fully discretized settings.
Load-bearing premise
The proof assumes that fitting the Gaussian model to discretized $X$ and a permuted, discretized copy of $Y$ gives a consistent estimate of the true cross-covariance $D_X^T P^T D_Y/(N-1)$, even though the permuted pairs are not i.i.d. draws from any joint Gaussian.
Editorial extensions
If this is right
- MPRT extends valid rank testing to all three data regimes: fully continuous, partially discretized, and fully discretized variables, with the continuous case as a special case.
- Used inside constraint-based causal discovery such as the PC algorithm, MPRT recovers skeletons more accurately than Fisher-Z or plug-in CCA tests when some variables are discretized, according to the paper's experiments.
- Fields that routinely collect ordinal data, such as psychometrics and biometrics, gain a rank test whose Type I error is controlled without needing continuous measurements.
- The validity holds for any discretization with at least two categories, and more categories improve power because less information is lost.
Reading between the lines
- A natural extension the paper does not pursue is replacing the Gaussian pseudo-likelihood with a nonparametric copula estimator, which could retain Type I error control when the parametric form of the underlying distribution is unknown.
- The load-bearing consistency step for permuted pairs could be tested directly: for a fixed permutation, compare the likelihood-based estimate of $D_X^T P^T D_Y/(N-1)$ to the actual sample quantity as $N$ grows; this isolates whether the discretized permutation trick is really delivering the claimed null distribution.
- If the permutation trick works for likelihood-based covariance estimates, the same construction may transfer to other consistent estimators of mixed-data correlation, such as rank-based or moment-based estimators, broadening the method's applicability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MPRT (Mixed data Permutation-based Rank Test), a permutation test for the rank of a cross-covariance matrix that is designed to remain valid when some or all variables are observed only through order-preserving discretization. The method first estimates the correlation matrix by a pairwise Gaussian pseudo-likelihood, computes CCA projections, and then estimates the null distribution of the rank statistic by randomly permuting one block of the data. The paper claims that the method controls Type I error in the presence of discretization, which existing CCA-based rank tests do not, and demonstrates the method on synthetic data, on the PC algorithm, and on a Big Five personality dataset. The theoretical support consists of an exchangeability result for the trailing canonical variables (Theorem 4) and a consistency result for the permutation statistic under discretization (Theorem 5).
Significance. If the theoretical claims were fully established, the paper would make a useful contribution: rank tests have applications in causal discovery, and a permutation-based version that handles discretized variables would address a real gap in the literature. The experimental results are consistent with the paper's Type I error claim, and the inclusion of real-data causal discovery is a strength. However, the central proof of Theorem 5 is not valid as written: it conflates consistency to a population quantity with the distributional approximation required for a permutation null. Because the abstract's headline claim of Type I error control rests on that theorem, the significance is currently conditional on a substantial proof repair.
major comments (3)
- [Appendix A.3, Theorem 5, Eq. (8)] The proof of Theorem 5 does not establish the claimed consistency. The final step asserts that \tilde D_X and P^T \tilde D_Y can be treated as samples from independent Gaussian vectors, and therefore that D_X^T P^T D_Y/(N-1) can be consistently estimated by the pseudo-likelihood estimator. This conflates two different targets. A consistent estimator of the cross-covariance under the permutation null converges to the population cross-covariance, which is zero, whereas the permutation statistic \lambda_k^P depends on the random sample cross-covariance whose fluctuations of order N^{-1/2} define the null distribution. Replacing the sample quantity by an estimate that converges to zero would make the permutation distribution degenerate and cannot yield a valid p-value. Moreover, the permuted pairs are not i.i.d. from a joint Gaussian: permutation without replacement introduces dependence across pairs, and when k>0 the full vectors X and Y are not independent even though their trailing canonical variables are. A valid proof would need to show that the estimated permutation statistic has the same limiting distribution as the statistic computed from the unobserved continuous permuted data, not merely that it converges to a population quantity.
- [Appendix A.1, Theorem 4] The proof of Theorem 4 invokes the continuous mapping theorem for SVD under the assumption of no repeated singular values. Under the null hypothesis H_k^0, however, the matrix \Sigma_X^{-1/2} \Sigma_{X,Y} \Sigma_Y^{-1/2} has K-k zero singular values, which are repeated whenever K-k>1. The singular vectors spanning the zero-singular-value subspace are not unique, so the claimed convergence A \to A_* and B \to B_* for the trailing columns is not justified. Since the statistic \lambda_k in Lemma 1 uses exactly those trailing canonical correlations, the paper needs to argue either that the test statistic is invariant to rotations within the zero-singular-value subspace or that the permutation null is consistently estimated despite the non-identifiability. This gap affects the validity of the permutation approach even in the all-continuous case.
- [Section 3.2, Theorem 5, Eq. (8)] The estimator in Eq. (8) uses A and B that are themselves estimated from the original data, but the proof treats them as fixed matrices. In the actual procedure, A and B are random and depend on the same data that are subsequently permuted, so the permutation distribution must account for this dependence. As stated, the consistency claim is not a well-defined asymptotic statement about the permutation statistic, because the probability space over which the convergence holds is not specified and the randomness of A and B is not separated from the randomness of the permutation.
minor comments (6)
- [Section 2.2] The section heading contains a typo: "Takses" should be "Takes".
- [Appendix A.3] The word "likeilhood" should be "likelihood".
- [Algorithm 1, line 8] Step 8 says to calculate \lambda_k^P following Eq. (4), but the statistic used for the permutation is defined in Eq. (6); this is inconsistent with the text and should be corrected.
- [Section 4.3] The phrase "cannot be not compared" is a double negative and should be rewritten, for example as "cannot be compared".
- [Figures 2 and 3] The Type I and Type II error plots do not include confidence intervals or standard error bars; with 3000 trials the standard error is small, but reporting it would make the comparison more informative.
- [Section 4.1] The description of threshold generation says "uniformly generate two thresholds from [-1.5,1.5]" but does not specify that the thresholds are sorted before discretization; please clarify.
Circularity Check
No significant circularity: the permutation null distribution is estimated empirically from permuted data, and the few self-citations are background rather than load-bearing.
full rationale
MPRT's central claim is that its permutation-based null distribution controls Type I error under discretization. The derivation chain is: (i) Theorem 4 establishes asymptotic independence of the tail canonical variables under the null, using joint Gaussianity and zero cross-covariance; this is a standard property, not an assumed conclusion. (ii) Lemma 1 is an algebraic identity rewriting the test statistic in terms of the tail canonical variables. (iii) Theorem 5 attempts to justify estimating the permutation statistic from discretized data via pseudo-likelihood. The estimation step is a separate statistical procedure whose consistency is asserted from standard pseudo-likelihood theory (Besag 1974; Gourieroux et al. 1984; Fan et al. 2017), not from the test's own target quantity. The permutation distribution is therefore built empirically from many random permutations, not derived from fitted constants or from the null hypothesis being tested. The paper does contain a genuine technical gap in the proof of Theorem 5: the claim that D_X^T P^T D_Y/(N-1) can be consistently estimated by maximizing likelihood on the permuted discretized data is not fully justified, because consistency for a population parameter does not automatically give the correct permutation distribution of a sample statistic. However, this is a correctness or completeness concern, not circularity: the estimator is not defined in terms of the p-value or the test decision, and no equation reduces to itself by construction. The self-citations (Dong et al. 2024a for rank-based causal discovery, Dong et al. 2024b for pseudo-likelihood parameterization) are background or experimental tools; the former is also supported by external results (Sullivant et al. 2010; Di 2009), and the latter is a standard pseudo-likelihood approach with external citations. No uniqueness theorem from the authors is used to force the choice of method, and no known empirical pattern is simply renamed. The paper also explicitly acknowledges in Appendix C.2 that it does not provide a theoretical power analysis, which is a stated limitation rather than a hidden circular assumption. Overall, the derivation is self-contained against external benchmarks and the modest self-citation load is not load-bearing; score 2 reflects minor non-load-bearing self-citation, not circularity.
Assumptions & free parameters
free parameters (1)
- thresholds T_j^t for discretized variables =
estimated via inverse Gaussian CDF (Eq. 14)
assumptions (4)
- domain assumption Underlying continuous variables are jointly Gaussian.
- standard math Pseudo-likelihood estimator is consistent for the mixed-data correlation matrix.
- standard math The normalized cross-covariance matrix has no repeated singular values.
- ad hoc to paper Permuted data can be treated as if they follow the Gaussian copula for consistent estimation of the cross-covariance.
Cite this review
Pith. "Pith review of Permutation-Based Rank Test in the Presence of Discretization and Application in Causal Discovery with Mixed Data." pith.science (2026). https://pith.science/paper/UCQWCW2B
@misc{pith2026250118990,
author = {Pith},
title = {Pith review of: Permutation-Based Rank Test in the Presence of Discretization and Application in Causal Discovery with Mixed Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/UCQWCW2B}},
note = {Machine review of arXiv:2501.18990}
}
read the original abstract
Recent advances have shown that statistical tests for the rank of cross-covariance matrices play an important role in causal discovery. These rank tests include partial correlation tests as special cases and provide further graphical information about latent variables. Existing rank tests typically assume that all the continuous variables can be perfectly measured, and yet, in practice many variables can only be measured after discretization. For example, in psychometric studies, the continuous level of certain personality dimensions of a person can only be measured after being discretized into order-preserving options such as disagree, neutral, and agree. Motivated by this, we propose Mixed data Permutation-based Rank Test (MPRT), which properly controls the statistical errors even when some or all variables are discretized. Theoretically, we establish the exchangeability and estimate the asymptotic null distribution by permutations; as a consequence, MPRT can effectively control the Type I error in the presence of discretization while previous methods cannot. Empirically, our method is validated by extensive experiments on synthetic data and real-world data to demonstrate its effectiveness as well as applicability in causal discovery.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Conditional Independence Tests for Constraint-Based Causal Discovery: A Survey
A survey that organizes conditional independence tests for constraint-based causal discovery into six families and connects test-level properties to graph-level errors.
Reference graph
Works this paper leans on
-
[1]
Anderson, T. W. An Introduction to Multivariate Statistical Analysis. 2nd ed. John Wiley & Sons, 1984
work page 1984
-
[2]
Partial correlation and conditional correlation as measure of conditional independence
Baba, K., Shibata, R., and Sibuya, M. Partial correlation and conditional correlation as measure of conditional independence. Australian and New Zealand Journal of Statistics, 46: 0 657--664, 12 2004
work page 2004
-
[3]
Spatial interaction and the statistical analysis of lattice systems
Besag, J. Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society: Series B (Methodological), 36 0 (2): 0 192--225, 1974
work page 1974
-
[4]
and Yongfeng, W
Changsheng, H. and Yongfeng, W. Investor sentiment and assets valuation. Systems Engineering Procedia, 3: 0 166--171, 2012
2012
-
[5]
H., Kalisch, M., and Richardson, T
Colombo, D., Maathuis, M. H., Kalisch, M., and Richardson, T. S. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pp.\ 294--321, 2012
2012
-
[6]
David, H. A. The beginnings of randomization tests. The American Statistician, 62 0 (1): 0 70--72, 2008
work page 2008
-
[7]
t-separation and d-separation for directed acyclic graphs
Di, Y. t-separation and d-separation for directed acyclic graphs. preprint, 2009
work page 2009
-
[8]
A versatile causal discovery framework to allow causally-related hidden variables
Dong, X., Huang, B., Ng, I., Song, X., Zheng, Y., Jin, S., Legaspi, R., Spirtes, P., and Zhang, K. A versatile causal discovery framework to allow causally-related hidden variables. In ICLR, 2024
work page 2024
Show all 42 references
-
[9]
A permutation-based kernel conditional independence test
Doran, G., Muandet, K., Zhang, K., and Sch \"o lkopf, B. A permutation-based kernel conditional independence test. In UAI, pp.\ 132--141, 2014
2014
-
[10]
Finney, D. J. Probit analysis: a statistical treatment of the sigmoid response curve. 1952
1952
-
[11]
Fisher, R. A. The distribution of the partial correlation coefficient. Metron, 3: 0 329--332, 1924
1924
-
[12]
probable error
Fisher, R. A. et al. 014: On the" probable error" of a coefficient of correlation deduced from a small sample. 1921
1921
-
[13]
Kernel measures of conditional dependence
Fukumizu, K., Gretton, A., Sun, X., and Sch\" o lkopf, B. Kernel measures of conditional dependence. In Advances in Neural Information Processing Systems, volume 20, 2007
2007
-
[14]
A kernel statistical test of independence
Gretton, A., Fukumizu, K., Teo, C., Song, L., Sch\" o lkopf, B., and Smola, A. A kernel statistical test of independence. In Advances in Neural Information Processing Systems, 2007
2007
-
[15]
Relations between two sets of variates
Hotelling, H. Relations between two sets of variates. In Breakthroughs in statistics: methodology and distribution, pp.\ 162--190. Springer, 1992
1992
-
[16]
Causal discovery from heterogeneous/nonstationary data
Huang, B., Zhang, K., Zhang, J., Ramsey, J., Sanchez-Romero, R., Glymour, C., and Sch \"o lkopf, B. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research, 21 0 (89): 0 1--53, 2020
2020
-
[17]
Huang, B., Low, C. J. H., Xie, F., Glymour, C., and Zhang, K. Latent hierarchical causal structure discovery with rank constraints. arXiv preprint arXiv:2210.01798, 2022
2022 arXiv
-
[18]
U., Ulvenes, P
Johnson, S. U., Ulvenes, P. G., ktedalen, T., and Hoffart, A. Psychometric properties of the general anxiety disorder 7-item (gad-7) scale in a heterogeneous psychiatric sample. Frontiers in psychology, 10: 0 1713, 2019
2019
-
[19]
Essai sur la g \'e om \'e trie \`a n dimensions
Jordan, C. Essai sur la g \'e om \'e trie \`a n dimensions. Bulletin de la Soci \'e t \'e math \'e matique de France , 3: 0 103--174, 1875
-
[20]
J \"o reskog, K. G. On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59 0 (3): 0 381--389, 1994
1994
-
[21]
and Friedman, N
Koller, D. and Friedman, N. Probabilistic graphical models: principles and techniques. MIT press, 2009
2009
-
[22]
Lord, F. M. and Novick, M. R. Statistical theories of mental test scores. IAP, 2008
2008
-
[23]
and Press, S
Nerlove, M. and Press, S. J. Univariate and multivariate log-linear and logistic models, volume 1306. Rand Corporation, 1973
1973
-
[24]
Maximum likelihood estimation of the polychoric correlation coefficient
Olsson, U. Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44 0 (4): 0 443--460, 1979
1979
-
[25]
Olsson, U., Drasgow, F., and Dorans, N. J. The polyserial correlation coefficient. Psychometrika, 47: 0 337--347, 1982
1982
-
[26]
Probabilistic reasoning in intelligent systems: Networks of plausible inference
Pearl, J. Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan kaufmann, 1988
1988
-
[27]
Pearl, J. et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19, 2000
2000
-
[28]
and Salmaso, L
Pesarin, F. and Salmaso, L. The permutation testing approach: a review. Statistica, 70 0 (4): 0 481--509, 2010
2010
-
[29]
A scalable conditional independence test for nonlinear, Non-Gaussian data
Ramsey, J. A scalable conditional independence test for nonlinear, Non-Gaussian data. arXiv preprint arXiv:1401.5031, 2014
2014 arXiv
-
[30]
A discovery algorithm for directed cyclic graphs
Richardson, T. A discovery algorithm for directed cyclic graphs. In Proceedings of the Twelfth International Conference on Uncertainty in Artificial Intelligence, 1996
1996
-
[31]
Rousseeuw, P. J. and Molenberghs, G. Transformation of non positive semidefinite correlation matrices. Communications in Statistics--Theory and Methods, 22 0 (4): 0 965--984, 1993
1993
-
[32]
Shah, R. D. and Peters, J. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Statistics, 2018
2018
-
[33]
Learning the structure of linear latent variable models
Silva, R., Scheine, R., Glymour, C., and Spirtes, P. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7 0 (Feb): 0 191--246, 2006
2006
-
[34]
Calculation of entailed rank constraints in partially non-linear and cyclic models
Spirtes, P. Calculation of entailed rank constraints in partially non-linear and cyclic models. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pp.\ 606--615. AUAI Press, 2013
2013
-
[35]
and Glymour, C
Spirtes, P. and Glymour, C. An algorithm for fast recovery of sparse causal graphs. Social Science Computer Review, 9: 0 62--72, 1991
1991
-
[36]
Causal inference in the presence of latent variables and selection bias
Spirtes, P., Meek, C., and Richardson, T. Causal inference in the presence of latent variables and selection bias. In Conference on Uncertainty in Artificial Intelligence, 1995
1995
-
[37]
N., Scheines, R., and Heckerman, D
Spirtes, P., Glymour, C. N., Scheines, R., and Heckerman, D. Causation, prediction, and search. MIT press, 2000
2000
-
[38]
Trek separation for gaussian graphical models
Sullivant, S., Talaska, K., and Draisma, J. Trek separation for gaussian graphical models. arXiv:0812.1938., 2010
1938 arXiv
-
[39]
Welch, W. J. Construction of permutation tests. Journal of the American Statistical Association, 85 0 (411): 0 693--698, 1990
1990
-
[40]
M., Renaud, O., Smith, S
Winkler, A. M., Renaud, O., Smith, S. M., and Nichols, T. E. Permutation inference for canonical correlation analysis. Neuroimage, 220: 0 117065, 2020
2020
-
[41]
Kernel-based conditional independence test and application in causal discovery
Zhang, K., Peters, J., Janzing, D., and Sch \"o lkopf, B. Kernel-based conditional independence test and application in causal discovery. In Conference on Uncertainty in Artificial Intelligence, 2012
2012
-
[42]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.