Motion estimation is the most important step in video compression. By using high precision motion vector in the H.264 encoder, the calculation is rapidly increased, but in the whole process of coding, motion estimation occupies about 80%. Although many motion estimation algorithms have been proposed to reduce the computational complexity of motion estimation, it still cannot meet the strict real-time demand. In this paper, based on the analysis of UMHexagonS algorithm, dynamic searching window is chosen in the UMHexagonS algorithm, then according to the motion activity, it uses different template to reduce the motion estimation time and improve video coding efficiency. Proved by the experiments on various test sequences, compared with the UMHexagonS algorithm, the motion estimation time of the proposed algorithm average saves 17.7525% in the case of the quality of the reconstructed image and rate close. It not only reduces the complexity of the algorithm, but also improves the real-time performance of the encoder.