The Mickens iteration method (MIM) is modified to solve self-excited systems containing nonsmooth nonlinearities and/or nonlinear damping terms. If the MIM is implemented routinely, the unknown frequency and amplitude of limit cycle (LC) would couple to each other in complicated nonlinear algebraic equations at each iteration. It is cumbersome to solve these algebraic equations, especially for nonsmooth systems. In the modified procedures, the unknown frequency is substituted by the determined value obtained at the previous iteration. By this means, the frequency is decoupled from the nonlinear terms. Numerical examples show that the LCs obtained by the modified MIM agree well with numerical results. The presented method is very suitable for solving self-excited systems, especially those with nonlinear damping and nonsmooth nonlinearities.