This paper presented a new improved Prony algorithm based on neural network to train weights.The algorithm solved some problems that difficulty and low precision during matrix inversion in Prony method. According to real-time transform characteristics of low frequency oscillation in power system, the algorithm used limited data windows in on-line parameter estimation and pattern recognition, and improved pattern recognition precision. The simulation results proved that this proposal algorithm has some features of directly ,effective, high reliability, less calculation amount and minor error when it be used to analysis oscillation characteristics and mode identification. So it is suitable for identification of low frequency oscillation mode in power system.