A tuned mass damper is a passive control device that has been widely used in aerospace, mechanical, and civil engineering as well as many other fields. Tuned mass dampers have been studied and improved over the course of many years. In practical engineering applications, a tuned mass damper inevitably produces some nonlinear characteristics due to the large displacement and the use of the limiting devices, but this nonlinearity is often neglected. The simulation results in this study confirm that neglecting the nonlinearity in the design process can produce adverse effects on the control performance. This paper takes into account the nonlinearity of the tuned mass damper produced in the process of vibration and deduces an optimum formula for the frequency of a tuned mass damper by the complexification averaging method and multiscale method. Based on this formula, a modified design method for the frequency of a tuned mass damper is presented. The numerical results show that the nonlinear tuned mass damper after modification is better than a linear tuned mass damper in terms of control performance.