The ball balancer has been used as a vibration suppression device in rotor systems. It has a superior characteristic that the vibration amplitude is reduced to zero theoretically at a rotational speed range higher than the critical speed. However, the ball balancer causes a self-excited vibration near the critical speed when the balls rotate in the balancer. This self-excited vibration may occur in the wide rotational speed range with a large amplitude vibration, and in such a case, escaping from it becomes difficult. In this paper, the occurrence region and the vibration characteristics of the self-excited vibration caused by the ball balancer are investigated. The nonlinear theoretical analysis is performed and a set of the fundamental equations governing the self-excited vibration is obtained. The influences of the parameters of the ball balancer, such as, the damping of the ball’s motion, the ball’s mass, and radius of the balls’ path, are explained and they are also validated experimentally.