Data collection plays an important role in pavement health monitoring, which is usually performed using costly devices, including point-based lasers and laser scanners. The main aim of this study measures pavement characteristics using an RGB-D sensor. By recording the depth and color images simultaneously, the sensor benefits the data fusion. By mounting the sensor on a moving cart, and fixing the vertical distance from the ground, data were collected along 100 m of the asphalt pavement using MATLAB. At each stop point, multiple frames were collected, the central region of interests was stored, and a low pass filter was subsequently applied to the data. To create a 3D surface of the pavement, sensor calibration was performed to map the RGB and depth infrared images. The SURF (speeded-up robust features) and MSAC (M-estimator sample consensus) algorithms were used to match the stitched images along the longitudinal profile. A case study of measuring roughness and rutting is applied to test the validity of the method. The result confirms that the proposed system is capable of measuring such indices with acceptable accuracy.