The design and implementation of video-decoding system are analyzed. And then, the video decoding and data processing scheme based on the FPGA and video-decoding chip ADV7181B is proposed. The I2C interface, used to configurate the register parameters of video-decoding chip ADV7181B, and the decoder of ITU-656, the storage of YCrCb, conversion between YCrCb and RGB and the display of VGA are implemented using FPGA. The all sub modules were described using Verilog HDL in the design, and merged in a module after they were emulated successfully. The module was come through and verified. The experiments results show that the system designed in this paper performed well.