Testing Clustered Equal Predictive Ability with Unknown Clusters

Alain Pirotte; Giovanni Urga; Oguzhan Akgun; Zhenlin Yang

REVIEW 3 major objections 5 minor 3 references

Testing Clustered Equal Predictive Ability with Unknown Clusters

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A selective inference procedure makes testing clustered equal predictive ability valid when clusters are estimated from the data.

desk verdict The selective-inference extension to panel k-means is genuinely new and the package is a plus, but the key independence lemma is not proven under general cross-sectional dependence and the simulations dodge that case. read the letter →

arxiv 2507.14621 v3 pith:TTEDFR73 submitted 2025-07-19 econ.EM

Oguzhan Akgun , Alain Pirotte , Giovanni Urga , Zhenlin Yang This is my paper · ORCID

classification econ.EM

keywords equalpredictiveabilitypaneldataselectiveinferenceKmeanspost-selectionp-valuecombinationforecastevaluationclusteredheterogeneity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to make it possible to test whether two forecasts are equally good on average within each of several clusters, when the clusters themselves are unknown and estimated from the same data by Panel Kmeans. That setup creates a double-dipping problem, and the paper argues that the right fix is selective conditional inference rather than sample splitting. The authors derive the asymptotic distribution of the square root of a Wald statistic for pairwise cluster equality conditional on the full sequence of cluster assignments produced by the algorithm, and show it is a truncated $\chi_P$ variable with a truncation set characterized by quadratic inequalities. They combine the resulting p-values for all pairs and for overall equal predictive ability using a merging function that controls family-wise Type I error under arbitrary dependence. The claimed consequence is that a researcher can test clustered equal predictive ability with unknown clusters and obtain asymptotically valid inference, supported by simulations and an exchange-rate forecasting application.

What carries the argument

The key object is the selective p-value $p[D_{k,g}(\hat C)]=1-F_{\chi_P}[d_{k,g}(\hat C);\mathcal T]$, where $F_{\chi_P}(\cdot;\mathcal T)$ is the CDF of a $\chi_P$ variable truncated to the set $\mathcal T$. The data are decomposed as $Z=\hat\Pi_{k,g}Z+D_{k,g}(\hat C)$T^{{-1/2}}$(\hat\nu_{k,g}/\|\hat\nu_{k,g}\|_2)\hat J_Z'\hat\Sigma_{k,g}^{1/2}(\hat C)$, separating the component that sets the statistic from components held fixed by conditioning. The conditioning event includes the cluster assignment at every iteration $$k_i^{{(m)}}$(Z)=$k_i^{{(m)}}$(z)$, which makes the truncation region a nested intersection of quadratic inequalities in the scalar $\phi$ that indexes the perturbation $z(\phi)$. This construction converts an intractable conditional distribution into a one-dimensional truncated chi-square calculation, and the same p-values are then aggregated with the generalized-mean merging function $F_{SI,r}$.

What would settle it

Rejection rates above the nominal level in a Monte Carlo design where the null holds and the same data set is clustered from many random starting points, with different seeds producing different final clusterings, would directly contradict the claimed uniform Type I error control. A concrete version: simulate the paper's AR(1) panel with $(\psi_1,\psi_2,\psi_3)=(0,0,0)$, run Algorithm 1 with two seeds that yield different assignment paths, and check whether the selective p-values for the same pair of clusters differ or whether the empirical rejection rate at $q=0.05$ exceeds 0.05.

Watch

Extended reading notes

Core claim

The central claim is that post-clustering inference on forecast loss differentials does not require sample splitting or known cluster memberships. Conditional on the event that every unit is assigned to the same cluster at every iteration of the Panel Kmeans algorithm, and conditional on nuisance directions, $D_{k,g}(\hat C)$, the square root of the pairwise Wald statistic, converges to a $\chi_P$ distribution truncated to the set $\mathcal T=\{\phi\ge 0:\cap_{m=1}^M\cap_{i=1}^N \{$k_i^{{(m)}}$(z(\phi))=$k_i^{{(m)}}$(z)\}\}$. The perturbation $z(\phi)$ moves the two estimated clusters toward or away from each other along the test direction, so the truncation set records which perturbations keep the clustering path unchanged. These inequalities are quadratic in $\phi$, making the selective p-value computable. Combining the $n_p$ pairwise p-values with the O-EPA p-value through the calibrated M-family merging function yields $F_{SI,r}$, and the paper's Theorem 3 states that under the null $\limsup P[p(F_{SI,r})\le q]\le q$ for every $q\in(0,1)$. Under well-separated alternatives with the true number of clusters, the test is consistent.

Load-bearing premise

The load-bearing premise is that the Panel Kmeans output, including the sequence of cluster assignments at every iteration, is a deterministic function of the data; the paper leaves a formal treatment of multiple random initializations to future work.

Editorial extensions

If this is right

A researcher can test clustered equal predictive ability with unknown clusters and obtain asymptotically valid inference without discarding data in a sample split.
The tests remain valid under general autocorrelation and cross-sectional dependence, including strong common-factor dependence, because the variance estimator is HAC-based.
When the number of clusters is chosen by the paper's information criterion, no additional conditioning is needed for the selective p-values to remain valid, provided the clustering output is unique.
If the alternative is true and the clusters are well separated, the combined test rejects with probability tending to one.
In the exchange-rate application, the selective test detects cluster-specific gains for machine-learning models such as XGBoost that an aggregate O-EPA test alone would not reveal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same conditioning-on-the-algorithm-path idea should transfer to other iterative clustering algorithms, such as hierarchical or spectral clustering, with the quadratic truncation sets replaced by the appropriate assignment rules.
Editorial inference: the paper's conjecture that multiple random initializations preserve validity is untested in the formal theory; a proof would likely require conditioning on the full initialization path or establishing uniqueness of the algorithm's output with probability one.
Editorial inference: the robustness simulation in the appendix suggests the selective test, unlike split-sample tests, keeps size control when cluster means change halfway through the sample, which points toward a formal theory of selective inference under structural breaks.
Editorial inference: the merging-function step is portable; any panel forecast comparison producing several dependent p-values on the same data could reuse the calibrated generalized-mean combination to preserve family-wise error control.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Good paper to know about, and a serious candidate for peer review, but the central proof has a gap that the authors have not acknowledged. The extension of selective inference from iid k-means to panel k-means is genuinely new, and the C-EPA testing framework is useful. They also ship working code and replication data, which is more than most.

The weak spot is Lemma S.7. The proof claims ΠZ, the test statistic, and the direction are asymptotically independent because Πν=0 and 'properties of the matrix normal distribution.' That step requires the covariance of vec(Z) to be scalar. Under G1–G3, which allow arbitrary autocorrelation and cross-sectional dependence, the projection and the contrast can be correlated through common factors. Without that independence, the truncation set in (14) is not the right conditioning, and the p-value in Proposition 1 may not control selective Type I error. I do not see this as a trivial technicality; it is the load-bearing step for Theorem 1 and Theorem 3.

The simulations don't help here. The Monte Carlo DGP uses a common factor with the same loading on every unit, so the contrast between cluster means cancels the factor. That design avoids the problematic dependence rather than testing it. The paper claims validity under 'arbitrary forms and strengths of cross-sectional dependence,' but the evidence doesn't cover the case where the Lemma S.7 argument fails.

The multiple-initialization issue is real and the authors are honest about it: they say a formal treatment is future work, and the proof of Proposition S.2 relies on a unique output. In practice they run 10 or 10000 starts. If the clustering depends on the seed, the conditioning event in Definition 1 is not a function of the data, so the selective p-value is not well-defined. They hand-wave that it controls error uniformly over initial partitions, but that is not a proof.

Everything else—the p-value merging, the OS variance estimator, the empirical application—is solid and clearly presented. The paper deserves a serious referee. My recommendation: send it to peer review, but ask for a major revision that either proves Lemma S.7 under the stated dependence assumptions or narrows the claims, and that addresses the initialization conditioning. As it stands, the central result is plausible but not proven.

Referee Report

3 major / 5 minor

Summary. The paper proposes a selective-inference framework for testing clustered equal predictive ability (C-EPA) when clusters are unknown and estimated by Panel Kmeans. The authors construct a Wald-type root statistic for pairwise equality of cluster means, claim that its asymptotic distribution conditional on the estimated clustering is a truncated χ variable, characterize the truncation set through quadratic inequalities, and combine the resulting pairwise p-values with an overall-EPA p-value using a dependence-robust merging function. Monte Carlo simulations compare the selective test with predetermined-cluster, naive, and split-sample tests, and an exchange-rate forecasting application illustrates the method.

Significance. If the central calibration result is correct, the paper makes a useful contribution: it offers a full-sample alternative to sample splitting for post-clustering forecast evaluation, accommodates conditioning variables, and is designed to be robust to autocorrelation and both weak and strong cross-sectional dependence. The p-value combination strategy is well motivated and the simulation study is extensive; the availability of replication code and R packages is a further strength. However, the validity of the selective p-values rests on an independence claim in Lemma S.7 that is not established under the paper's general dependence assumptions, and the implementation's reliance on random initializations is acknowledged as an unresolved issue. These are load-bearing gaps for the main theorem.

major comments (3)

[Online Appendix H.3, Lemma S.7] The proof claims that Z'ν_{k,g} and Π_{k,g}Z are independent because Π_{k,g}ν_{k,g}=0 and by 'properties of the matrix normal distribution.' This is valid only when vec(Z) has scalar covariance, or more generally when ν_{k,g} is an eigenvector of Cov(Z). Assumptions G1–G3 allow arbitrary autocorrelation and cross-sectional dependence. For example, if Z_{it}=λ_i f_t+v_{it} with heterogeneous loadings λ_i, the scalar Z'ν_{k,g} contains the factor average weighted by the cluster loading difference, while Π_{k,g}Z retains the factor component because the factor vector is not orthogonal to the complement unless λ_i is constant across units. The two quantities are then correlated, so the conditional law of D_{k,g} given Π_{k,g}Z=π is not the unconditional χ_P law truncated to the set T. This invalidates the derivation of the p-value in Proposition 1 and the proof of Theorem 1(a) under the stated general dependence assumptions. The Monte Carlo design in Section 5.1 uses a common factor with the same loading λ for all units, so the contrast D_{k,g} is orthogonal to the factor and the problematic dependence is not exercised.
[Section 3.1, Definition 1; Section I, Algorithm 1; Online Appendix D] The conditioning event in Definition 1 and the truncation set T in (14) require that the Panel Kmeans output, including the sequence of assignments at every iteration, be a deterministic function of the data. Algorithm 1 as stated is not deterministic: it starts from random initializations, and Section 5.1 reports that all tests use 10 random initializations. Section D explicitly states that 'does using multiple initializations require additional conditioning? Intuitively, no' and that 'a formal treatment is left for future work, but our simulations support this conjecture.' With random starts, the realized output is not a deterministic function of Z, so the event ∩_{i=1}^N {k_i(Z)=k_i(z)} is not well-defined for the implemented procedure. This is a load-bearing gap because the reported simulations and the provided R package rely on multiple initializations, yet the theoretical guarantee in Theorem 1(a) applies only to a deterministic clustering map.
[Online Appendix H.1, Lemma 1(b)] The proof asserts that after noting a linear combination inherits mixing properties, 'Corollary 2.2 of Phillips & Durlauf (1986) applies, yielding a multivariate invariance principle.' That citation concerns time-series invariance principles for integrated processes and does not by itself cover N^{1-ε}-scaled cross-sectional averages under strong cross-sectional dependence, which is the case ε=1 highlighted in the paper. The conditions needed for the Gaussian limit with the stated normalizing matrix, especially under factor structures with nonzero average loadings, are not verified. Please provide a self-contained argument or a precise theorem that covers the strong-dependence case, or state the additional assumptions required for Lemma 1(b).

minor comments (5)

[Theorem 3(a)] The statement 'lim sup p(F_SI,r)≤q' appears to omit the probability operator; it should read 'lim sup P[p(F_SI,r)≤q]≤q'.
[Online Appendix H.3, Lemma S.6] The phrase 'for all k,g∈{2,...,K}, k≠g' should presumably be 'for all k,g∈{1,...,K}, k≠g'.
[Equations (12)–(13)] The notation for the covariance estimator is inconsistent: equation (12) uses S-hat^{-1/2}_{k,g}(C) in J_Z while the main text and equation (13) use Σ-hat^{-1/2}_{k,g}(C); please use a single symbol.
[Section I, Algorithm 1] The input line 'Z=(Z'_{11},Z'_{12}...,Z')′' appears truncated; it should be 'Z=(Z'_{11},Z'_{12},...,Z'_{NT})′'.
[Acknowledgements] There is a typo: 'Galatararay University' should be 'Galatasaray University'.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction found; selective p-values are derived conditioning quantities, with minor self-citations and two self-flagged mathematical gaps that are correctness risks rather than circularity.

full rationale

The paper's central result is a conditioning computation, not a fitted prediction. Proposition 1 gives p[dk,g(C)] = 1 - F_chi_P[dk,g(C); T], with T defined as the set of phi preserving all iteration-wise cluster assignments; this is a truncated-chi survival function, and no parameter is calibrated to make it match the data. The C-EPA p-merging in Theorem 3 uses the external validity theorem of Vovk & Wang (2020) (Proposition 5) for the normalization constant, and the O-EPA statistic follows Sun (2013); the self-citations to APUY and Spreng & Urga (2023) are comparative remarks and are not load-bearing. The tuning choices (r in [-inf,-1), zeta=1.5) are simulation-calibrated, but the validity theorems hold for the whole stated range of r, and the IC penalty is an auxiliary estimator, so no fitted quantity is renamed as a prediction. The paper itself flags two genuine limitations, which are correctness risks rather than circularity: (i) in OA Section D after Proposition S.2, 'does using multiple initializations require additional conditioning? Intuitively, no. ... A formal treatment is left for future work, but our simulations support this conjecture' - so the conditioning event of Definition 1 is not proven to be a deterministic function of the data; (ii) Lemma S.7's proof of asymptotic independence of D and the projection rests on 'properties of the matrix normal distribution', which is not established under the arbitrary cross-sectional dependence allowed by Assumptions G1-G3; the Monte Carlo design uses a common factor with the same loading for all units, so this dependence is not exercised. These gaps weaken the force of the theoretical claim but do not make the derivation equivalent to its inputs; the selective p-value is derived, not fitted.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard panel assumptions (G1-G3, S1-S3), plus two assumptions that are not fully established: an invariance principle for strong cross-sectional dependence and uniqueness of the k-means output. Tuning parameters r and zeta are free choices validated by simulation. No new physical or conceptual entities are introduced.

free parameters (3)

r = not stated (chosen in [-infinity,-1), selected via simulation)
Exponent in the p-value merging function F_SI,r; the paper states simulation experiments favored a particular choice within this range, but no single value is reported in the main text. Validity holds for any r in the range, but finite-sample size depends on it.
zeta (IC penalty) = 1.5
Tuning parameter in the information criterion for selecting K; set to 1.5 to tolerate mild overestimation, based on simulation performance in the range [1.5,3].
B (number of cosines) = min(floor(P T^(2/3)), T) in simulations
Bandwidth for the orthonormal series long-run variance estimator; theory requires B to infinity with B/T to 0, so the simulation rule is a practical choice, not a fitted constant. Included for completeness.

assumptions (6)

domain assumption Assumptions G1-G3: finite moments, stationary mixing errors, positive definite long-run variance, dependence conditions.
Invoked in Lemma 1, Proposition 1, Theorems 1-3 for CLT and variance estimation.
domain assumption Assumptions S1-S3: true cluster structure with well-separated centers, exponential tail bounds, and mixing for cluster membership consistency.
Used in Lemma 2 and for power consistency of the tests.
domain assumption Assumption SS for the Split Sample comparator: Vit independent of measurable E_{t-l} variables beyond lag l.
Restricts time dependence for validity of the split-sample test, which is compared but not central to the proposed method.
ad hoc to paper Multivariate invariance principle for N^{1-epsilon} scaled cross-sectional averages under strong CD.
In the proof of Lemma 1(b), invariance is asserted by citing Phillips & Durlauf (1986) without verifying conditions for strong cross-sectional dependence; the scaling with epsilon is introduced to unify weak and strong CD.
ad hoc to paper Uniqueness of the Panel Kmeans output and deterministic mapping from data to cluster assignments.
Required for the conditioning set in Definition 1 and for Proposition S.2. The paper notes Algorithm 1 does not guarantee uniqueness and leaves the formal treatment of multiple initializations to future work.
standard math Validity of the Vovk-Wang p-value merging function under arbitrary dependence.
Used to construct F_SI,r and to claim family-wise error control; taken from external theorems in Vovk & Wang (2020) and Vovk et al. (2022).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Testing Clustered Equal Predictive Ability with Unknown Clusters." pith.science (2026). https://pith.science/paper/TTEDFR73

@misc{pith2026250714621,
  author       = {Pith},
  title        = {Pith review of: Testing Clustered Equal Predictive Ability with Unknown Clusters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTEDFR73}},
  note         = {Machine review of arXiv:2507.14621}
}

abstract

We develop tests of clustered equal predictive ability (C-EPA) in panels where the clusters are unknown and estimated by the Panel Kmeans algorithm. To address the challenge of testing hypotheses that depend on data-driven clusters, we adopt a selective conditional inference framework. Specifically, we first derive a Wald-type test for pairwise equality and show that the limiting distribution of its square root conditional on the estimated clusters is that of a truncated $\chi$ variable. We characterize the associated truncation set by quadratic inequalities in the data space. Then, for the C-EPA hypothesis, we propose a $p$-value combination method by aggregating the evidence against the pairwise equality and overall EPA null hypotheses. The Monte Carlo results show accurate size control and good finite-sample power of the proposed tests. An empirical application to exchange-rate forecasting, using both traditional time-series models and machine-learning methods, illustrates the practical relevance of our procedure.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages

[1]

& Yang, Z

Akgun, O., Pirotte, A., Urga, G. & Yang, Z. (2024), ‘Equal predictive ability tests based on panel data with applications to OECD and IMF forecasts’,International Journal of Forecasting40(1), 202–228. Ando, T. & Bai, J. (2017), ‘Clustering huge number of financial time series: A panel data approach with high-dimensional predictors and factor structures’,J...

work page 2024
[656]

Consistent Selection of the Number of Groups in Panel Models via Cross-Validation

Hillebrand, E., Mikkelsen, J. G., Spreng, L. & Urga, G. (2023), ‘Exchange rates and macroeconomic fundamentals: Evidence of instabilities from time-varying factor loadings’, Journal of Applied Econometrics 38(6), 857–877. Hoga, Y. & Dimitriadis, T. (2023), ‘On testing equal conditional predictive ability under measurement error’, Journal of Business & Eco...

work page Pith review arXiv 2023
[808]

Selective inference for multiple pairs of clusters after K-means clustering

Welch, I. & Goyal, A. (2008), ‘A comprehensive look at the empirical performance of equity premium prediction’, Review of Financial Studies 21(4), 1455–1508. West, K. D. (1996), ‘Asymptotic inference about predictive ability’, Econometrica 64(5), 1067–1084. Yun, Y. & He, Y. (2024), ‘Selective inference for multiple pairs of clusters after k-means clusteri...

work page Pith review arXiv 2008

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

& Yang, Z

Akgun, O., Pirotte, A., Urga, G. & Yang, Z. (2024), ‘Equal predictive ability tests based on panel data with applications to OECD and IMF forecasts’,International Journal of Forecasting40(1), 202–228. Ando, T. & Bai, J. (2017), ‘Clustering huge number of financial time series: A panel data approach with high-dimensional predictors and factor structures’,J...

work page 2024

[2] [656]

Consistent Selection of the Number of Groups in Panel Models via Cross-Validation

Hillebrand, E., Mikkelsen, J. G., Spreng, L. & Urga, G. (2023), ‘Exchange rates and macroeconomic fundamentals: Evidence of instabilities from time-varying factor loadings’, Journal of Applied Econometrics 38(6), 857–877. Hoga, Y. & Dimitriadis, T. (2023), ‘On testing equal conditional predictive ability under measurement error’, Journal of Business & Eco...

work page Pith review arXiv 2023

[3] [808]

Selective inference for multiple pairs of clusters after K-means clustering

Welch, I. & Goyal, A. (2008), ‘A comprehensive look at the empirical performance of equity premium prediction’, Review of Financial Studies 21(4), 1455–1508. West, K. D. (1996), ‘Asymptotic inference about predictive ability’, Econometrica 64(5), 1067–1084. Yun, Y. & He, Y. (2024), ‘Selective inference for multiple pairs of clusters after k-means clusteri...

work page Pith review arXiv 2008