BACKGROUND: The frequencies that can evoke strong steady state visual evoked potentials (SSVEP) are limited, which leads to brain-computer interface (BCI) instruction limitation in the current SSVEP-BCI. To solve this problem, the visual stimulus signal modulated by trinary frequency shift keying was introduced. OBJECTIVE: The main purpose of this paper is to find a more reliable recognition algorithm for SSVEP-BCI based on trinary frequency shift keying modulated stimuli. METHODS: First, the signal modulated by trinary frequency shift keying is simulated by MATLAB. At different noise levels, the empirical mode decomposition, singular value decomposition, and synchrosqueezing with the short-time Fourier transform are used to extract the characteristic frequency and reconstruct the signal. Then, the coherent method is used to demodulate the reconstructed signal. Second, in the paradigm of BCI using trinary frequency shift keying modulated stimuli, the three methods mentioned above are used to reconstruct EEG signals, and canonical correlation analysis and coherent demodulation are used to recognize the BCI instructions. RESULTS: For simulated signals, it is found that synchrosqueezing with short-time Fourier transform has a better effect on extracting the characteristic frequencies. For the EEG signal, it is found that the method combining synchrosqueezing with short-time Fourier transform and coherent demodulation has a higher accuracy and information translate rate than other methods. CONCLUSION: The method combining synchrosqueezing with short-time Fourier transform and coherent demodulation proposed in this paper can be applied in the SSVEP system based on trinary frequency shift keying modulated stimuli.