In view of the characteristics of the non-stationary and multi-component AM-FM signals of vibration signals in the rolling element bearing, the generalized demodulation time-frequency analysis method is used for its fault diagnosis, overcoming the problem that the maximal overlap discrete wavelet packet transform (MODWPT) has no adaptability. First of all, the original vibration signal is took preprocessing by generalized Fourier; Then, using MODWPT to decompose signals after pretreatment and obtaining weights; Once again, the weights are carried out the inverse generalized Fourier transform to get the weights of the original signal; Finally, reconstructing principal component of the original signal to get the Hilbert instantaneous energy spectrum, roller bearing fault diagnoses based on the Hilbert instantaneous energy spectrum. The experimental results show that the method can effectively diagnose rolling bearing fault.