The paper proposes an improved adaptive Gaussian mixture model (GMM) approach with online EM algorithms for updating, which solves the video segmentation problems carried by busy environment and illumination change. Different learning rates are set for foreground district and background district respectively, which improves the convergence speed of background model. A shadow removal scheme is also introduced for extracting complete moving objects. It is based on brightness distortion and chromaticity distortion in RGB color space. Morphological filtering and connected components analysis algorithm are also introduced to process the result of background subtraction. The experiment results show that the improved GMM has good accuracy and high adaptability in video segmentation. It can extract a complete and clear moving object when it is incorporated with shadow removal.