The information hidden in an image is worth more than a thousand words. Proper analysis of a medical image can help in timely detection and diagnose of a disease which increases the rate of survival of cancer patients. Analysis of images manually is subjective and time consuming. On
the other hand, automated analysis of a medical image has a lot of challenges due to the architecture and colors of the medical images. This paper, gives a survey on detection, classification and diagnosis of colorectal cancer and proposes a deep learning based techniques to differentiate
between healthy tissues and cancerous polyps in histology images. It also compares the accuracy of three different classification frameworks namely Convolutional Neural Network (CNN), Fully Convolutional Network (FCN) and Recurrent Neural Network (RNN). It also presents the overview of the
work done in this field. It first discusses basic deep learning methods and then the known techniques used for detection, classification and diagnosis of colorectal cancer followed by the comparative analysis of all the surveyed paper. Finally, it talks about the conclusion, challenges and
the future scope of the progress in this field.