pith. sign in

arxiv: 2501.02746 · v2 · submitted 2025-01-06 · 📡 eess.SP · math.PR· math.SP· math.ST· stat.TH

A Large-Dimensional Analysis of ESPRIT DoA Estimation: Inconsistency and a Correction via RMT

Zhengyu Wang , Wei Yang , Xiaoyi Mai , Zenan ling , Zhenyu Liao , Robert C. Qiu This is my paper

Pith reviewed 2026-05-23 06:30 UTC · model grok-4.3

classification 📡 eess.SP math.PRmath.SPmath.STstat.TH
keywords ESPRITdirection-of-arrival estimationrandom matrix theorylarge-dimensional analysissample covariance matrixinconsistencyconsistency correction
0
0 comments X

The pith

Classical ESPRIT produces inconsistent direction-of-arrival estimates when array size and snapshot count grow large together, while a random-matrix-theory correction called G-ESPRIT restores consistency.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that the standard ESPRIT algorithm, which depends on the sample covariance matrix, yields inconsistent direction-of-arrival estimates in the regime where the number of sensors N and snapshots T both tend to infinity with their ratio approaching a positive constant. This failure occurs for both widely spaced and closely spaced sources because the sample covariance matrix itself becomes an inconsistent estimator of the population covariance in high dimensions. The authors introduce G-ESPRIT, a modified procedure that incorporates random matrix theory corrections to the subspace estimation step, and prove that the resulting estimates remain consistent under the same asymptotic conditions. A supporting technical result is a new bound on the difference of eigenvalues of two possibly non-Hermitian matrices. Readers should care because many practical array-processing systems operate precisely in this large-dimensional regime, where the classical method's bias can produce systematically wrong angle readings.

Core claim

In the large-dimensional regime where N and T tend to infinity with N/T approaching c in (0, infinity), the classical ESPRIT algorithm relying on the sample covariance matrix produces inconsistent DoA estimates for both widely- and closely-spaced directions of arrival. The proposed G-ESPRIT method, derived via random matrix theory, is proven to be consistent in the same setting.

What carries the argument

The G-ESPRIT algorithm, which replaces the direct use of the sample covariance matrix in the ESPRIT subspace step with a random-matrix-theory correction that accounts for the large-dimensional bias.

If this is right

  • G-ESPRIT DoA estimates converge to the true angles even when the number of sensors and snapshots increase at the same rate.
  • The consistency result covers both widely spaced and closely spaced sources.
  • The new eigenvalue-difference bound for non-Hermitian matrices can be applied in other subspace or perturbation analyses.
  • Numerical experiments match the predicted inconsistency of classical ESPRIT and the consistency of G-ESPRIT.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Other subspace-based estimators that rely on the sample covariance matrix are likely to exhibit similar inconsistency when applied to large arrays.
  • Practical systems with growing sensor counts may require adoption of random-matrix corrections to avoid accumulating angle errors.
  • The approach could be tested on real array data to check whether the asymptotic correction improves finite-sample performance.

Load-bearing premise

The novel bound on eigenvalue differences between two potentially non-Hermitian matrices holds and correctly links the random matrix analysis to the ESPRIT subspace computation.

What would settle it

Monte Carlo trials in which N and T increase while keeping N/T fixed, showing that the classical ESPRIT root-mean-square error stays away from zero while the G-ESPRIT error approaches zero.

Figures

Figures reproduced from arXiv: 2501.02746 by Robert C. Qiu, Wei Yang, Xiaoyi Mai, Zenan ling, Zhengyu Wang, Zhenyu Liao.

Figure 1
Figure 1. Figure 1: System diagram for DoA estimation. A far-field [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MSEs of classical ESPRIT as a function of subarrary [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Approximation errors in spectral norm versus array [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Left: comparison between DoA estimates ˆθ from ESPRIT in blue, ¯θ from Theorem 2 in green, ˆθ G from G-ESPRIT of Proposition 1 in red, and true DoAs θ in purple; in the same setting as the right plot of [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Left: comparison between DoA estimates ˆθ from ESPRIT in blue, ¯θ from Theorem 2 in green, ˆθ G from G-ESPRIT of Proposition 1 in red, and true DoAs θ in purple; in the same setting as the right plot of [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Empirical MSEs for closely-spaced DoA (with [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
read the original abstract

In this paper, we perform asymptotic analyses of the widely used ESPRIT direction-of-arrival (DoA) estimator for large arrays, where the array size $N$ and the number of snapshots $T$ grow to infinity at the same pace. In this large-dimensional regime, the sample covariance matrix (SCM) is known to be a poor eigenspectral estimator of the population covariance. We show that the classical ESPRIT algorithm, that relies on the SCM, and as a consequence of the large-dimensional inconsistency of the SCM, produces inconsistent DoA estimates as $N,T \to \infty$ with $N/T \to c \in (0,\infty)$, for both widely-~and~closely-spaced DoAs. Leveraging tools from random matrix theory (RMT), we propose an improved G-ESPRIT method and prove its consistency in the same large-dimensional setting. From a technical perspective, we derive a novel bound on the eigenvalue differences between two potentially non-Hermitian matrices, which may be of independent interest. Numerical simulations are provided to corroborate our theoretical findings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript performs a large-dimensional asymptotic analysis (N, T → ∞ with N/T → c ∈ (0, ∞)) of the ESPRIT DoA estimator. It shows that classical ESPRIT, which relies on the sample covariance matrix (SCM), produces inconsistent DoA estimates for both widely- and closely-spaced sources due to the known large-dimensional inconsistency of the SCM. The authors introduce a G-ESPRIT variant that applies random matrix theory (RMT) corrections and prove its consistency in the same regime; the proof invokes a novel bound on eigenvalue differences between two potentially non-Hermitian matrices to connect the corrected covariance to the ESPRIT rotational-invariance step. Numerical simulations are used to support the claims.

Significance. If the central claims hold, the work supplies a rigorous demonstration that classical ESPRIT fails in modern large-array regimes and supplies a corrected estimator whose consistency is established under the same scaling. The novel eigenvalue-difference bound may be of independent interest beyond DoA estimation. The paper applies established RMT tools to a concrete signal-processing problem and derives a new technical lemma to close the argument.

major comments (2)
  1. [Section containing the novel bound and its application to G-ESPRIT consistency (likely the technical core following theR] The consistency proof for G-ESPRIT (the central positive claim) rests on the novel bound for |λ_i(A) − λ_i(B)| between potentially non-Hermitian matrices. This bound is invoked to link the RMT-corrected covariance to the subspace step; because the bound is derived specifically for the present setting and has no prior literature support, its proof and the precise conditions under which it applies to the matrices arising in G-ESPRIT must be verified before the consistency result can be accepted.
  2. [Asymptotic analysis of classical ESPRIT] The inconsistency result for classical ESPRIT is load-bearing for the motivation of G-ESPRIT. The derivation should explicitly trace how the SCM eigenvalue bias propagates through the signal-subspace extraction and the rotational-invariance equation to the final DoA estimates; a reference to the precise equations or lemmas that close this chain would strengthen the argument.
minor comments (2)
  1. [Abstract] The abstract states that simulations corroborate the findings but does not indicate the range of c values or the array sizes N used; adding this information would help readers assess the practical relevance of the large-dimensional regime.
  2. [Preliminaries / Notation] Notation for the population and sample covariance matrices, as well as the definition of the G-ESPRIT correction, should be introduced with explicit equation numbers at first use to improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading, positive assessment of significance, and constructive major comments. We address each point below and will incorporate clarifications to strengthen the manuscript.

read point-by-point responses

  1. Referee: The consistency proof for G-ESPRIT (the central positive claim) rests on the novel bound for |λ_i(A) − λ_i(B)| between potentially non-Hermitian matrices. This bound is invoked to link the RMT-corrected covariance to the subspace step; because the bound is derived specifically for the present setting and has no prior literature support, its proof and the precise conditions under which it applies to the matrices arising in G-ESPRIT must be verified before the consistency result can be accepted.

    Authors: We thank the referee for highlighting the technical core. The proof of the novel eigenvalue-difference bound appears in full in Appendix B, derived under the paper's large-dimensional assumptions (N,T→∞, N/T→c, bounded source powers, and the specific perturbation structure from the RMT-corrected covariance). The bound applies directly to the pair of matrices formed by the corrected sample covariance and its population counterpart in the rotational-invariance step. To facilitate verification, we will add a short remark in Section IV explicitly restating the assumptions used in the bound and confirming they hold for the G-ESPRIT matrices; no change to the result itself is required. revision: yes

  2. Referee: The inconsistency result for classical ESPRIT is load-bearing for the motivation of G-ESPRIT. The derivation should explicitly trace how the SCM eigenvalue bias propagates through the signal-subspace extraction and the rotational-invariance equation to the final DoA estimates; a reference to the precise equations or lemmas that close this chain would strengthen the argument.

    Authors: We agree that an explicit propagation trace will improve clarity. In the revised manuscript we will insert a new paragraph immediately after Theorem 1 in Section III that walks through the chain: (i) the known SCM eigenvalue bias (Lemma 2, citing the RMT result of [ref]) perturbs the signal-subspace eigenvectors via a Davis-Kahan-type bound (Lemma 3); (ii) this perturbed subspace enters the rotational-invariance equation (Eq. (12)); (iii) the resulting bias in the estimated rotation matrix propagates to the DoA estimates via the arctangent mapping (Eq. (13)). We will add forward references to these lemmas and equations. revision: yes

Circularity Check

0 steps flagged

No circularity: RMT analysis and novel bound are independent derivations

full rationale

The paper uses established RMT results on SCM inconsistency to demonstrate classical ESPRIT failure, then derives a new eigenvalue gap bound for non-Hermitian matrices to prove G-ESPRIT consistency. Neither step reduces to a fitted parameter renamed as prediction, a self-definitional loop, or a load-bearing self-citation chain. The novel bound is presented as an original technical contribution derived in the paper itself, making the overall argument self-contained rather than circular by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Central claims rest on the large-dimensional regime N,T → ∞ with N/T → c and the applicability of random matrix theory to the sample covariance and ESPRIT eigenvalue problem; no free parameters or invented entities are indicated in the abstract.

axioms (1)
  • domain assumption Standard random matrix theory assumptions on the sample covariance matrix in the proportional growth regime N/T → c
    The paper leverages RMT tools which rely on these asymptotic regimes for the inconsistency analysis.

pith-pipeline@v0.9.0 · 5753 in / 1295 out tokens · 55692 ms · 2026-05-23T06:30:39.732202+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages

  1. [1]

    Multiple emitter location and signal pa rameter esti- mation,

    Ralph Schmidt, “Multiple emitter location and signal pa rameter esti- mation,” IEEE Trans Antennas Propag. , vol. 34, no. 3, pp. 276–280, 1986

    work page 1986

  2. [2]

    Performance analysis of roo t-music,

    Bhaskar D Rao and KV Sl Hari, “Performance analysis of roo t-music,” IEEE Trans Acoust. , vol. 37, no. 12, pp. 1939–1949, 1989

    work page 1939

  3. [3]

    A subspace rotation a pproach to signal parameter estimation,

    A. Paulraj, R. Roy, and T. Kailath, “A subspace rotation a pproach to signal parameter estimation,” Proc. IEEE, vol. 74, no. 7, pp. 1044–1046, 1986

    work page 1986

  4. [4]

    Music, maximum likeliho od, and cramer-rao bound,

    Petre Stoica and Arye Nehorai, “Music, maximum likeliho od, and cramer-rao bound,” IEEE Trans Acoust. , vol. 37, no. 5, pp. 720–741, 1989

    work page 1989

  5. [5]

    Perf ormance analysis of the total least squares esprit algorithm,

    Bjorn Ottersten, Mats Viberg, and Thomas Kailath, “Perf ormance analysis of the total least squares esprit algorithm,” IEEE Trans Signal Process., vol. 39, no. 5, pp. 1122–1135, 1991

    work page 1991

  6. [6]

    Sensor array processi ng based on subspace fitting,

    Mats Viberg and Bjorn Ottersten, “Sensor array processi ng based on subspace fitting,” IEEE Trans Signal Process. , vol. 39, no. 5, pp. 1110– 1121, 1991

    work page 1991

  7. [7]

    On the resolution performance of spectral analysis,

    Petre Stoica, Virginija ˇSimonyte, and Torsten S¨ oderstr¨ om, “On the resolution performance of spectral analysis,” Signal processing , vol. 44, no. 2, pp. 153–161, 1995

    work page 1995

  8. [8]

    Random matrix theory i n statistics: A review,

    Debashis Paul and Alexander Aue, “Random matrix theory i n statistics: A review,” J Stat Plan Inference. , vol. 150, pp. 1–29, July 2014

    work page 2014

  9. [9]

    Romain Couillet and Zhenyu Liao, Random Matrix Methods for Machine Learning , Cambridge University Press, 2022

    work page 2022

  10. [10]

    Performance analysis of an improved MUSIC DoA estimator,

    Pascal V allet, Xavier Mestre, and Philippe Loubaton, “ Performance analysis of an improved MUSIC DoA estimator,” IEEE Trans Signal Process., vol. 63, no. 23, pp. 6407–6422, 2015

    work page 2015

  11. [11]

    Modified subspace al- gorithms for DoA estimation with large arrays,

    Xavier Mestre and Miguel ´Angel Lagunas, “Modified subspace al- gorithms for DoA estimation with large arrays,” IEEE Trans Signal Process., vol. 56, no. 2, pp. 598–614, 2008

    work page 2008

  12. [12]

    Improved subspace estimation for multivariate observations of high dimension: the deterministic signals case,

    Pascal V allet, Philippe Loubaton, and Xavier Mestre, “ Improved subspace estimation for multivariate observations of high dimension: the deterministic signals case,” IEEE Trans Inf Theory. , vol. 58, no. 2, pp. 1043–1068, 2012

    work page 2012

  13. [13]

    Large information plus noise random matrix models and consistent subspace estimation in large sensor network s,

    Walid Hachem, Philippe Loubaton, Xavier Mestre, Jamal Najim, and Pascal V allet, “Large information plus noise random matrix models and consistent subspace estimation in large sensor network s,” Random Matrices: Theory and Applications , vol. 1, no. 02, pp. 1150006, 2012

    work page 2012

  14. [14]

    A clt for the g-music doa estimator,

    Pascal V allet, Xavier Mestre, and Philippe Loubaton, “ A clt for the g-music doa estimator,” Proc. European Signal Processing Conference (EUSIPCO), pp. 2298–2302, 2012

    work page 2012

  15. [15]

    Asymptotic analysis of a consistent subspace estimator fo r observations of increasing dimension,

    Xavier Mestre, Pascal V allet, Philippe Loubaton, and W alid Hachem, “Asymptotic analysis of a consistent subspace estimator fo r observations of increasing dimension,” pp. 677–680, 07 2011

    work page 2011

  16. [16]

    A subspace rotation approach to signal parameter estimation,

    A. Paulraj, R. Roy, and T. Kailath, “A subspace rotation approach to signal parameter estimation,” Proc. IEEE, vol. 74, no. 7, pp. 1044–1046, 1986

    work page 1986

  17. [17]

    Comparative perfor mance of ESPRIT and MUSIC for direction-of-arrival estimation,

    R. Roy, A. Paulraj, and T. Kailath, “Comparative perfor mance of ESPRIT and MUSIC for direction-of-arrival estimation,” in ICASSP ’87. IEEE International Conference on Acoustics, Speech, a nd Signal Processing, Apr. 1987, vol. 12, pp. 2344–2347

    work page 1987

  18. [18]

    Performanc e Analysis of MUSIC and ESPRIT DOA Estimation Algorithms for Adaptive Array Smart Antenna in Mobile Communication,

    T.B. Lavate, V .K. Kokate, and A.M. Sapkal, “Performanc e Analysis of MUSIC and ESPRIT DOA Estimation Algorithms for Adaptive Array Smart Antenna in Mobile Communication,” in 2010 Second International Conference on Computer and Network Technolo gy, Apr. 2010, pp. 308–311

    work page 2010

  19. [19]

    On the Performan ce of MUSIC With Toeplitz Rectification in the Context of Large Arrays,

    Pascal V allet and Philippe Loubaton, “On the Performan ce of MUSIC With Toeplitz Rectification in the Context of Large Arrays,” IEEE Trans Signal Process. , vol. 65, no. 22, pp. 5848–5859, Nov. 2017

    work page 2017

  20. [20]

    V an der V aart, Asymptotic Statistics, vol

    Aad W. V an der V aart, Asymptotic Statistics, vol. 3 of Cambridge Series in Statistical and Probabilistic Mathematics , Cambridge University Press, 2000

    work page 2000

  21. [21]

    Esprit-estimation of signal par ameters via rotational invariance techniques,

    R. Roy and T. Kailath, “Esprit-estimation of signal par ameters via rotational invariance techniques,” IEEE Trans Acoust. , vol. 37, no. 7, pp. 984–995, 1989

    work page 1989

  22. [22]

    Det erministic equivalents for certain functionals of large random matric es,

    Walid Hachem, Philippe Loubaton, and Jamal Najim, “Det erministic equivalents for certain functionals of large random matric es,” Ann Appl Probab., vol. 17, no. 3, pp. 875–930, 2007

    work page 2007

  23. [23]

    Y u. A. Rozanov, Stationary Random Processes , Holden-Day series in time series analysis. Holden-Day, San Francisco, 1967

    work page 1967

  24. [24]

    Dis tribution of eigenvalues for some sets of random matrices,

    Vladimir A Marcenko and Leonid Andreevich Pastur, “Dis tribution of eigenvalues for some sets of random matrices,” Mathematics of the USSR-Sbornik, vol. 1, no. 4, pp. 457, 1967

    work page 1967

  25. [25]

    On the Empirical D istribution of Eigenvalues of a Class of Large Dimensional Random Matric es,

    Jack W. Silverstein and Zhidong Bai, “On the Empirical D istribution of Eigenvalues of a Class of Large Dimensional Random Matric es,” J Multivar Anal. , vol. 54, no. 2, pp. 175–192, 1995

    work page 1995

  26. [26]

    Eigenvalues of lar ge sample covariance matrices of spiked population models,

    Jinho Baik and Jack W. Silverstein, “Eigenvalues of lar ge sample covariance matrices of spiked population models,” J Multivar Anal. , vol. 97, no. 6, pp. 1382–1408, 2006

    work page 2006

  27. [27]

    The e igenvalues and eigenvectors of finite, low rank perturbations of large rand om matrices,

    Florent Benaych-Georges and Raj Rao Nadakuditi, “The e igenvalues and eigenvectors of finite, low rank perturbations of large rand om matrices,” Adv Math (N Y). , vol. 227, no. 1, pp. 494–521, 2011

    work page 2011

  28. [28]

    38–42, Cambridge university press, 2012

    Roger A Horn and Charles R Johnson, Matrix analysis , pp. 38–42, Cambridge university press, 2012

    work page 2012

  29. [29]

    8–11, Courier Corporation, 2012

    William Raymond Scott, Group theory, pp. 8–11, Courier Corporation, 2012

    work page 2012

  30. [30]

    Continuity of th e roots of a polynomial,

    Melvyn B. Nathanson and David A. Ross, “Continuity of th e roots of a polynomial,” Commun Algebra. , vol. 52, no. 6, pp. 2509–2518, June 2024

    work page 2024

  31. [31]

    P erformance analysis of esprit-type algorithms for non-circular sourc es,

    Jens Steinwandt, Florian Roemer, and Martin Haardt, “P erformance analysis of esprit-type algorithms for non-circular sourc es,” IEEE International Conference on Acoustics, Speech and Signal P rocessing, pp. 3986–3990, 2013. 12 APPENDIX A TECHNICAL LEMMAS In this section, we prepare the readers with a few technical l emmas and examples. Lemma 3 (Woodbury ...

    work page 2013

  32. [32]

    in the case of widely-spaced DoAs in Assumption 3 that, ∥AHA − IK∥ =O(N − 1), (37) and ∥AHJH 1 J1A − τIK ∥ =O(N − 1) ∥AHJH 1 J2A − τ diag{eı∆θi}K i=1∥ =O(N − 1), (38) with τ = limn/N ∈ (0, 1), so that the steering matrix A is (approximately for N large) the same as UK, the top- K subspace of C, and that both AHJH 1 J1A and AHJH 1 J2A are asymptotically di...

    work page

  33. [33]

    in the case of closely-spaced DoAs in Assumption 4 with K = 2 sources such that θ2 =θ1 +α/N for α> 0, AHA = [ 1 eıα/ 2 sinc( α 2 ) e− ıα/ 2 sinc( α 2 ) 1 ] +O∥·∥(N − 1), where sinc(t) = sin(t)/t , and AHJH 1 J1A = [ τ 1− eıατ − ıα 1− e− ıατ ıα τ ] +O∥·∥(N − 1), AHJH 1 J2A = [ τeı∆θ1 1− eıατ − ıα eı∆θ2 1− e− ıατ ıα eı∆θ1 τeı∆θ2 ] +O∥·∥(N − 1), (39) so that...

    work page

  34. [34]

    Proof of Lemma 5

    (40) In particular , the approximation errors of both eigenvalue s and eigenvectors (in an Euclidean norm sense) are of order O(N − 1), by W eyl’s inequality and Davis–Kahan theorem, respective ly. Proof of Lemma 5. In the case of widely-spaced DoAs under Assumption 3, we have , per the definition of a(θi) in (2) and of J1, J2 in (6), that a(θi)Ha(θj) = 1 ...

    work page

  35. [35]

    approximation (in a spectral norm sense) of the expectati on of the random matrix model of interest; and

    work page

  36. [36]

    Here, we provide detailed derivation of the first step for the results in Lemma 7, the second concentration step is rather s tandard, see [9, Chapter 2]

    concentration of trace and bilinear norms as in Definition 3 around the corresponding expectations. Here, we provide detailed derivation of the first step for the results in Lemma 7, the second concentration step is rather s tandard, see [9, Chapter 2]. We evaluate the expectation 1 T E[ZHQ(z1)JT 1 J1Q(z2)Z], and that of 1 T ZHQ(z1)JT 1 J2Q(z2)Z can be deri...

    work page