In recent years, deep learning based networks have achieved good performance in brain tumour segmentation of MR Image. Among the existing networks, U-Net has been successfully applied. In this paper, it is propose deep-learning based Bidirectional Convolutional LSTM XNet (BConvLSTMXNet) for segmentation of brain tumor and using GoogLeNet classify tumor & non-tumor. Evaluated on BRATS-2019 data-set and the results are obtained for classification of tumor and non-tumor with Accuracy: 0.91, Precision: 0.95, Recall: 1.00 & F1-Score: 0.92. Similarly for segmentation of brain tumor obtained Accuracy: 0.99, Specificity: 0.98, Sensitivity: 0.91, Precision: 0.91 & F1-Score: 0.88.