Based on the establishment of angular contact ball bearing mechanical model, a nonlinear coupled lateral, torsional and axial dynamic model of helical gear-rotor-bearing system is established, and the dynamic differential equations of the coupled lateral-torsional-axial nonlinear vibration are deduced for imbalance rotors. The investigations are systematically carried out by oscillograms and spectrograms with rotating speed, taking into account eccentricity and nonlinear supporting by rolling bearing. The results show that the rotation frequency of the driven shaft appears in the driving shaft. In addition, the rotation frequencies and meshing frequency appear obviously in torsional direction. It can be seen that the lateral, torsional and axial response characteristics of driving and driven shafts obvious differences are due to the effects of the gear assembly characteristic, gear geometry parameters and the angular contact ball bearings characteristics. As a result, not only appear the rotational frequency and stiffness frequency, but also yield the bearing variable stiffness frequency and conbined frequency in lateral directions. However, the theory of the helical gear-rotor-bearing system still needs further research.