This paper introduced about the in-process vibration testing method for small diameter endmill. By this method, the natural frequency and modal parameters such as mass, damping, and stiffness of the milling tool can be determined in the milling process. An oscillation of the vibrator is controlled by the function generator to apply the impact force at the appropriate cutting period. The measurement setup can determine the compliance curve by the measurement signals of the exiting force and tool deformation. To evaluate the feasibility of the new method, vibration tests were performed on a square endmill which has the diameter of 4 mm in the milling on brass material. Results of vibration tests show that modal parameters of the specific vibration mode can be determined by the new developed method.