In recent years, the Face recognition task has been an active research area in computer vision and biometrics. Many feature extraction and classification algorithms are proposed to perform face recognition. However, the former usually suffer from the wide variations in face images, while the latter usually discard the local facial features, which are proven to be important for face recognition. In this paper, a novel framework based on merging the advantages of the Key points Local Binary/Tetra Pattern (KP-LTrP) and Improved Hough Transform (IHT) with the Improved DragonFly Algorithm-Kernel Ensemble Learning Machine (IDFA-KELM) is proposed to address the face recognition problem in unconstrained conditions. Initially, the face images are collected from the publicly available dataset. Then noises in the input image are removed by performing preprocessing using Adaptive Kuwahara filter (AKF). After preprocessing, the face from the preprocessed image is detected using the Tree-Structured Part Model (TSPM) structure. Then, features, such as KP-LTrP, and IHT are extracted from the detected face and the extracted feature is reduced using the Information gain based Kernel Principal Component Analysis (IG-KPCA) algorithm. Then, finally, these reduced features are inputted to IDFA-KELM for performing FR. The outcomes of the proposed method are examined and contrasted with the other existing techniques to confirm that the proposed IDFA-KELM detects human faces efficiently from the input images.