<div>The key benefits of cloud services, such as low cost, access flexibility, and mobility, have attracted users worldwide to utilize the deep learning algorithms for developing computer vision tasks. Untrusted third parties maintain these cloud servers, and users are always concerned about sharing their confidential data with them. In this paper, we addressed these concerns for by developing SecureDL, a privacy-preserving image recognition model for encrypted data over cloud. Additionally, we proposed a block-based image encryption scheme to protect images’ visual information. The scheme constitutes an order-preserving permutation ordered binary number system and pseudo-random matrices. The encryption scheme is proved to be secure in a probabilistic viewpoint and through various cryptographic attacks. Experiments are performed for several image recognition datasets, and the achieved recognition accuracy for encrypted data is close with non-encrypted data. SecureDL overcomes the storage, and computational overheads occurred in fully-homomorphic and multi-party computations based secure recognition schemes. </div>