We analyze modulation instability (MI) in lossy metamaterials (MMs) with both nonlinear polarization and nonlinear magnetization. A detailed discussion on the role of the nonlinear electric polarization, nonlinear magnetic susceptibility and linear loss on the MI is presented. It is found that the nonlinear magnetization and linear loss play an important role in determining the MI in nonlinear MMs. Linear stability analysis predicts that MI may occur in the MMs with self-defocusing nonlinearity due to the negative-index of MMs. In particular, the nonlinear magnetic susceptibility changes the generation condition of MI seriously, it can be used to enhance or depress the MI depending on the sign of the nonlinear magnetic susceptibility. Finally, the influence of the loss in MMs on the MI is discussed.
Modulation instability of structured-light beams in negative-index metamaterials