This paper focuses on integrating information from RGB and thermal infrared modalities to perform RGB-T object tracking in the correlation filter framework. Our baseline tracker is Staple (Sum of Template and Pixel-wise LEarners), which combines complementary cues in the correlation filter framework with high efficiency. Given the input RGB and thermal videos, we utilize the baseline tracker due to its high performance in both of accuracy and speed. Different from previous correlation filter-based methods, we perform the fusion tracking at both the pixel-fusion and decision-fusion levels. Our tracker is robust to the dataset challenges, and due to the efficiency of FFT, our tracker can maintain high efficiency with superior performance. Extensive experiments on the RGBT234 dataset have demonstrated the effectiveness of our work.