Based on the definition of fractional derivative, the paper proposed a unique new idea to describe the viscoelastic property of asphalt mixture with fractional calculus. According to the SPT (Simple Performance Tests) test results, the dynamic modulus and phase angle of asphalt mixture were determined. The result of the test was fitted with the classical Kelvin model, the Maxwell model, the solid model with three elements, respectively. It showed that the classical viscoelastic model did not simulate the dynamic mechanical properties of asphalt mixture properly. Since the existing constitutive relation cannot describe well the dynamic viscoelastic properties of asphalt mixture, the fractional derivative viscoelastic model with three elements was adopted and its fitting effect analyzed. The result shown a good fitting for the fractional derivative viscoelastic model with three elements, and a few test parameters were required to build the mode. In addition, these simulating parameters were significant in physics. The order of the fractional derivative has good correlation with the phase angle, incarnating the viscoelastic proportion of asphalt mixture. So the fractional derivative viscoelastic model with three elements can accurately describe the dynamic mechanical properties of asphalt mixture.