The amino acid sequence of a protein contains all the necessary information to specify its shape, which dictates its biological activities. However, it is challenging and expensive to experimentally determine the three-dimensional structure of proteins. The backbone torsion angles, as an important structural constraint, play a critical role in protein structure prediction, and accurately predicting the angles can considerably advance the tertiary structure prediction by accelerating efficient sampling of the large conformational space for low energy structures. On account of the rapid growth of protein databases and striking breakthroughs in deep learning algorithms, computational advances allow us to extract knowledge from large-scale data to address key biological questions. Here we propose evolutionary signatures that are computed from protein sequence profiles, and a deep neural network, termed ESIDEN, that adopts a straightforward architecture of recurrent neural networks with a small number of learnable parameters. The proposed ESIDEN is validated on three benchmark datasets, including D2020, TEST2016/2018, and CASPs datasets. On the D2020, using the combination of the four novel features and basic features, the ESIDEN achieves the mean absolute error (MAE) of 15.8 and 20.1 for ϕ and ψ, respectively. Comparing to the best-so-far methods, we show that the ESIDEN significantly improves the angle ψ by the MAE decrements of more than 2 degrees on both TEST2016 and TEST2018 and achieves closely approximate MAE of the angle ϕ although it adopts simple architecture and fewer learnable parameters. On fifty-nine template-free modeling targets, the ESIDEN achieves high accuracy by reducing the MAEs by about 0.4 and more than 2.5 degrees on average for the torsion angles ϕ and ψ in the CASPs, respectively. Using the predicted torsion angles, we infer the tertiary structures of four representative template-free modeling targets that achieve high precision with regard to the root-mean-square deviation and TM-score by comparing them to the native structures. The results demonstrate that the ESIDEN can make accurate predictions of the torsion angles by leveraging the evolutionary signatures compared to widely used classical features. The proposed evolutionary signatures would be also used as alternative features in predicting residue-residue distance, protein structure, and protein-ligand binding sites. Moreover, the high-precision torsion angles predicted by the ESIDEN can be used to accurately infer protein tertiary structures, and the ESIDEN would potentially pave the way to improve protein structure prediction.