A vision-based method for detecting the cracks in the concrete sleepers of the railway tracks will be introduced in this paper. The method is able to detect and partially classify the cracks of the concrete sleepers in two successive steps based on the image processing and pattern recognition techniques. The method has been implemented on the acquired image data frames followed by the analysis, experimental, comparison results and evaluation. The presented results are reasonable which indicates the goodness of the introduced method. The preliminary results of this work have been presented in [A. Delforouzi, A. H. Tabatabaei, M. H. Khan and M. Grzegorzek, A vision-based method for automatic crack detection in railway sleepers, in Kurzynski, M., Wozniak, M., Burduk, R. (eds.), Proceedings of the 10th International Conference on Computer Recognition Systems CORES 2017, Polanica Zdroj, Poland. CORES 2017. Advances in Intelligent Systems and Computing, Vol. 578 (Springer, Cham, 2018), pp. 130–139, doi: 10.1007/978-3-319-59162-9_14].