This paper considers the problem of detecting a few signals in high-dimensional complex-valued Gaussian data satisfying Johnstone's [On the distribution of the largest eigenvalue in principal components analysis, Ann. Statist.29 (2001) 295–327] spiked covariance model. We focus on the difficult case where signals are weak in the sense that the sizes of the corresponding covariance spikes are below the phase transition threshold studied in Baik et al. [Phase transition of the largest eigenvalue for non-null complex sample covariance matrices, Ann. Probab.33 (2005) 1643–1697]. In contrast to the majority of previous studies, we base the signal detection on the information contained in all the eigenvalues of the sample covariance matrix, as opposed to a few of the largest ones. This allows us to detect weak signals with non-trivial asymptotic probability when the dimensionality of the data and the number of observations go to infinity proportionally. We derive a simple analytical expression for the maximal possible asymptotic probability of correct detection holding the asymptotic probability of false detection fixed. To accomplish this derivation, we establish a novel representation for the hypergeometric function [Formula: see text] of two p × p matrix arguments, one of which has a deficient rank r < p, as a repeated contour integral of the hypergeometric function [Formula: see text] of two r × r matrix arguments.
Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices
