In high-precision machining, it is very important to obtain operational modal parameters of machine tools to ensure the quality of machined parts. More specifically, the prediction of free chatter boundary is a critical requirement for improving productivity, which requires accurate dynamic parameters under operational conditions. Operational modal analysis is an effective method to estimate dynamic parameters of the tested system. However, for machine tools, the direct application of the operational modal analysis method is not feasible. In measured responses, periodic components are very preponderant during cutting, which is mainly caused by the periodic cutting force. In order to overcome this issue and to extract operational modal parameters of machine tools, the method of time-synchronous-subtraction based on the principle of time-synchronization-averaging is proposed. The correctness of the proposed approach is demonstrated by a simulation experiment, and the feasibility is verified through a milling experiment. The results show that the time-synchronous-subtraction method can well eliminate harmonics in the vibration response under the real cutting condition. The method is not only accurate but also simple in calculation.