With the FPGA technology progressing, the speed, the internal multiplier and the internal RAM of the FPGA are increasing. Its internal resource can be allocated flexibility, and there is no limit on the pipeline stages, so it is more suitable for real-time video processing comparing with the previous DSP and PC. By this reason, the DE2 development system is selected as the real-time video processing platform, which has a core of the Cyclone II series FPGA, in which the calculation of LK-algorithm-based real-time optical flow is implemented. Finally, by the reasonable overall arrangement for the pipeline and the sub-pipeline, the system achieves the real-time video motion tracking for the 640×480 resolution 30 frames/s images.