To effectively find the direction of non-circular signals received by a uniform linear array (ULA) in the presence of non-negligible perturbations between array elements, i.e., mutual coupling, in colored noise, a direction of arrival (DOA) estimation approach in the context of high order statistics is proposed in this correspondence. Exploiting the non-circularity hidden behind a certain class of wireless communication signals to build up an augmented cumulant matrix, and carrying out a reformulation of the distorted steering vector to extract the angular information from the unknown mutual coupling, by exploiting the characteristic of mutual coupling, i.e., a limited operating range and an inverse relation of coupling effects to interspace, we develop a MUSIC-like estimator based on the rank-reduction (RARE) technique to directly determine directions of incident signals without mutual coupling compensation. Besides, we provide a solution to the problem of coherency between signals and mutual coupling between sensors co-existing, by selecting a middle sub-array to mitigate the undesirable effects and exploiting the rotation-invariant property to blindly separate the coherent signals into different groups to enhance the degrees of freedom. Compared with the existing robust DOA methods to the unknown mutual coupling under the framework of fourth-order cumulants (FOC), our work takes advantage of the larger virtual array and is able to resolve more signals due to greater degrees of freedom. Additionally, as the effective aperture is virtually extended, the developed estimator can achieve better performance under scenarios with high degree of mutual coupling between two sensors. Simulation results demonstrate the validity and efficiency of the proposed method.
AR Model-Based Direction-of-Arrival Estimation of Coherent Signals in the Presence of Unknown Mutual Coupling