Although bladed disks are nominally designed to be cyclically symmetric (tuned system), the vibration characteristics of all the blades on a disk are slightly different due to the manufacturing tolerance, deviations in the material properties, and wear during operation. These small variations break the cyclic symmetry and split the eigenvalue pairs. Bladed disks with small variations are referred to as a mistuned system. Many researchers suggest that while mistuning has an undesirable effect on the forced response, it has a beneficial (stabilizing) effect on blade flutter (the self-excited vibration). Therefore, it is necessary to optimize a bladed disk for forced vibration and blade flutter. In this study, firstly, the stability analysis of a mistuned bladed disk of a steam turbine that experienced the blade flutter in the field is carried out by use of the reduced order model, the Fundamental Mistuning Model. It is reported that the bladed disk analyzed failed due to unstalled flutter of the 1st mode, and the problem was solved by alternating mistuning. By comparing the analysis results with these field experiences, the analysis method is validated. Secondly, a parametric study on the mistuning effect is carried out for typical mistuning patterns, such as periodic and random mistuning, for both forced and self-excited vibrations. Finally, based on the above-mentioned results, a practical optimization method considering both forced vibration and self-excited vibration with respect to the bladed disk of a steam turbine with a free-standing blade structure is proposed.