This paper proposes the UmNet model based on convolutional neutral network (CNN), aiming to improve the ability to recognize and classify concrete cracks in a background complicated by construction seams and seepage traces. The model was derived from the famous CNN AlexNet. Without changing the receptive field, large convolutional kernels were replaced with small ones to reduce the parameters, deepen the network, and increase nonlinear transforms. Next, convolutional block attention module (CBAM) was introduced to highlight the key information in images and focus on high-weight channels. Finally, Bayesian network (BN) layer and L2 regularization were added, and the number of nodes in fully connected layer were reduced. A series of comparative experiments were carried out on three datasets D, P, and W. The results show that the proposed UmNet surpassed AlexNet in the recognition accuracy on D, P, and W by 3.74%, 3.17%, and 5.74%, respectively, and reduced the number of parameters by 75.04%. Therefore, our model is an effective means to recognize and classify of concrete cracks under strong interference.