A self-calibration method of coupling error is presented for 3-DOF displacement measurement of a planar moving stage based on two planar gratings. The self-calibration method using Fourier series is developed to extract the periodic systematic errors from the coupling errors. The extracted periodic systematic errors are compensated. Experiments are conducted to validate the validity of the self-calibration method and experimental results indicate that the coupling errors in x and y directions are reduced by 2 and 1.5 times respectively. It can be therefore concluded that the self-calibration method is suitable for the 3-DOF displacement measurement of a planar moving stage to improve the positioning accuracy.