This paper proposes a nearby phase search (NPS) algorithm based on BPS estimation algorithm in optical coherent receivers. And its suitable for arbitrary multi-level modulation. Making use of the continuity of phase change, the proposed NPS algorithm is applied to process nearby symbols by taking the pre-estimation phase of each symbol block as reference point. Compared to the traditional blind phase search (BPS) algorithm and its improved two-stage BPS algorithm, the performance of the proposed NPS algorithm is greatly improved in ultra-high speed coherent optical transmission system. By the simulation, the effectiveness and feasibility of the proposed algorithm are demonstrated in 28GBaud 16-QAM and 64-QAM system. Its shown that the computational complexity of the NPS algorithm greatly reduces in the guarantee of laser line width tolerance and bit error rate.