In order to overcome the problem that the single-Gauss model is poor of anti-interference and Gaussian Mixture model is poor of real-time, we present the double modeling algorithm of moving target detection. We use three frame difference method to distinguish the invariant region and complex region in background. And then we use single-Gauss modeling to model the invariant background while the complex region of background would be modeled with Gaussian Mixture modeling. It is more effective than the single-Gauss model and more efficient than the Gaussian Mixture model .The experimental results show that the improved algorithm is superior to the traditional single Gauss model or Gaussian Mixture model. It can detect moving target more quickly and accurately, with good robustness and real-time.