<p>Most of the smoke detection system these days still using sensors that have to receive specific particles before it could give a warning. But, this system takes some time to react and quite difficult to place in spacious room or the outdoor. To overcome this, there is some research that build smoke detection system using many kind video processing technique that could provide early warning. In this research, wavelet energy was used to detect smoke in the video. To determine candidate blocks in a frame that contain smoke, this research performed background subtraction and color analysis based on HSV color space. Then implementing spatial analysis and spatio-temporal analysis by using wavelet energy method and accumulative motion orientation to detect the smoke. This system using combination of dataset from previous research [1], downloaded from various sources and self-made dataset. Based on testing process using those dataset, this system reaches 91.05% accuracy for block-level and 72.22% accuracy for frame-level.</p><strong>Keywords: </strong>Accumulative motion orientation, smoke detection, spatial analysis, spatio-temporal analysis, video processing, wavelet energy