Residual displacement spectrum is one of the most important means to predict the permanent deformation of structures after the earthquake, and various normalizations of residual displacements have generally been used for construction of the spectrum. However, the issue regarding the merits and drawbacks of each normalization has not yet been investigated thoroughly. A comparison between two normalizations that relate the residual displacements to the elastic and inelastic displacements is made in terms of the effect of ground motion and structural characteristics by means of the results of nonlinear time history analysis. The statistical results reveal that the residual-to-peak-inelastic displacement ratios have the advantages of small dispersion, samples without any outliers, and relatively symmetric distribution, which benefits from the strong correlation between residual and peak inelastic displacements. Moreover, the residual-to-peak-inelastic displacement ratios are almost independent of site conditions, significant duration, and natural periods. Consequently, the peak inelastic displacements are superior to the elastic ones as an intermediate step for residual displacements estimation, provided that the peak inelastic displacements are estimated with a low uncertainty. For providing alternatives to estimate residual displacement demands, the constant-strength residual displacement spectra are developed for both normalizations.