The resolution of structural finite element model (FEM) determines the computation cost and accuracy in dynamic analysis. This study proposes a novel wavelet finite element model (WFEM), which facilitates adaptive mesh refinement, for the dynamic analysis and damage detection of beam structures subjected to a moving load (ML). The multi-scale equations of motion for the beam under the ML are derived using the second-generation cubic Hermite multi-wavelets as the shape functions. Then an adaptive-scale analysis strategy is established, in which the scales of the wavelet beam elements are dynamically changed according to the ML position. The performance of the multi-scale WFEM is examined in both dynamic analysis and damage detection problems. It is demonstrated that the multi-scale WFEM with a similar number of degrees of freedom can achieve much higher accuracy than the traditional FEM. In particular, the multi-scale WFEM enables the detection of sub-element damage with a progressive model updating process. The advantage in computation efficiency and accuracy makes the proposed method a promising tool for multi-scale dynamic analysis or damage detection of structures.