Cervical cancer is the second most common cancer affecting women worldwide but is curable if diagnosed early. Routinely, expert pathologists visually examine histology slides for assessing cervix tissue abnormalities. A localized, fusion-based, hybrid imaging and deep learning approach is explored to classify squamous epithelium into cervical intraepithelial neoplasia (CIN) grades for a dataset of 83 digitized histology images. Partitioning the epithelium region into 10 vertical segments, 27 handcrafted image features and rectangular patch, sliding window-based convolutional neural network features are computed for each segment. The imaging and deep learning patch features are combined and used as inputs to a secondary classifier for individual segment and whole epithelium classification. The hybrid method achieved a 15.51% and 11.66% improvement over the deep learning and imaging approaches alone, respectively, with a 80.72% whole epithelium CIN classification accuracy, showing the enhanced epithelium CIN classification potential of fusing image and deep learning features.