On the Resolution Probability of Conditional and Unconditional Maximum Likelihood DoA Estimation

After decades of research in Direction of Arrival (DoA) estimation, today Maximum Likelihood (ML) algorithms still provide the best performance in terms of resolution capabilities. At the cost of a multidimensional search, ML algorithms achieve a significant reduction of the outlier production mechanism in the threshold region, where the number of snapshots per antenna and/or the signal to noise ratio (SNR) are low. The objective of this paper is to characterize the resolution capabilities of ML algorithms in the threshold region. Both conditional and unconditional versions of the ML algorithms are investigated in the asymptotic regime where both the number of antennas and the number of snapshots are large but comparable in magnitude. By using random matrix theory techniques, the finite dimensional distributions of both cost functions are shown to be Gaussian distributed in this asymptotic regime, and a closed form expression of the corresponding asymptotic covariance matrices is provided. These results allow to characterize the asymptotic behavior of the resolution probability, which is defined as the probability that the cost function evaluated at the true DoAs is smaller than the values that it takes at the positions of the other asymptotic local minima.