In order to meet the trend and consumer demands for video information, the ISO/IEC group and ITU-T video encoding expert group have cooperated in making the new video encoding standard of HEVC. It defines 35 Intra prediction modes and thus has a very high encoding complexity. In order to reduce this complexity, this paper has proposed a new algorithm to effectively reduce the number of the candidate mode which needs to be checked. The edge detection and Hough transform are utilized for the prediction unit (PU) with different sizes. Statistical analysis is utilized for the detected edge line angles to decide the candidate modes that need to be checked. The C++ and OpenCV language have been utilized for the implementation of the proposed algorithm. The proposed algorithm can reduce the encoding time by 43.72 percent at most and 17.06 percent at least with just little increase of the code rate and small decrease of the PSNR.
