The existing object tracking method using covariance modeling is hard to reach the desired tracking performance when the deformation of moving target and illumination changes are drastic, we proposed a object tracking algorithm based on bilateral filtering. Firstly, the algorithm deals the image to be tracked with bilateral filtering, and extracts the needed features of filtered image to construct covariance matrix as tracking model. Secondly, under log-Euclidean Riemannian metric, we construct similarity measure for object covariance matrix and model updating strategy. Extensive experiments show that the proposed method has better adaptability for object deformation and illumination changes.