This paper aims to implement an distributed intelligent video analysis system based on TI multimedia Digital Signal Processor (DSP) TMS320DM642, at the same time, the algorithm optimization is also described. The intelligent video analysis system we proposed provides users with fast and precise video analysis services. The video image is transmitted to the image processing board by analog channels and IP cameras, then the DSP processing the video flow. In this system, target detection, segmentation, feature extraction, alarm and automatic tracking can be implemented. In this system, several optimization techniques are also used in algorithms, including the algorithm level optimization (ALO), the program level optimization (PLO) and the instruction level optimization (ILO). Experimental results show the exellent result in reducing the CPU load.