In this article, we focused on the super-resolution technique in computer vision applications. During last decade, image super-resolution techniques have been introduced and adopted widely in various applications. However, increasing demand of high quality multimedia data has lead towards the high resolution data streaming. The conventional techniques which are based on the image super-resolution are not suitable for multi-frame SR. Moreover, the motion estimation, motion compensation, spatial and temporal information extraction are the well-known challenging issues in video super-resolution field. In this work, we address these issues and developed deep learning based novel architecture which performs feature alignment, filtering the image using deep learning and estimates the residual of low-resolution frames to generate the high-resolution frame. The proposed approach is named as Align-Filter & Learn Video Super resolution using Deep learning (AFLVSR). We have conducted and extensive experimental analysis which shows a significant improvement in the performance when compared with the state-of-art video SR techniques.