Traditional median filter algorithm has the long processing time, which goes against the real-time image processing. According to its shortcomings, this paper puts forward the rapid median filter algorithm, and uses DE2 board of the company called Altera to do the realization on FPGA (CycloneII 2C35). The experimental results show that the image pre-processing system is able to complete a variety of high-level image algorithms in milliseconds, and FPGA's parallel processing capability and pipeline operations can dramatically improve the speed of image processing, so the FPGA-based image processing system has broad prospects for development.