The sliding of the rolling element in the load zone would cause the bearing’s wear and failure at high speed under elastohydrodynamic lubrication (EHL) condition. Aiming at this phenomenon, considering lubrication oil film, time-varying displacement, radial clearance, and comprehensive stiffness, a five degree-of-freedom (DOF) dynamic model of rolling bearing with local defect is proposed based on isothermal EHL and which is validated by experimental data. The variation of oil film stiffness, comprehensive stiffness, and vibration characteristics of rolling bearing is studied under different speeds and loads. The results show that the lubricating oils with different viscosities have a certain influence on the bearing oil film thickness and comprehensive stiffness. As the load increases, the oil film stiffness and comprehensive stiffness would increase, and the oil film thickness would decrease. And as the tangential speed increases, the oil film stiffness would increase, and the oil film thickness and comprehensive stiffness would decrease. The vibration amplitude of the rolling bearing is enhanced with the increase of the rotation speed and the radial load. This model is helpful for the optimization, the correct use of lubricants, and life prediction of rolling bearing.