Handwritten numeral recognition method generally uses neural networks, the more prominent of these is BP neural network, but BP algorithm is easily get in a local minimum of the error-prone and causes slow oscillation and training , general solution for it is to optimize the structure of the algorithms first. Therefore, on the basis of the analysis of GA-BP algorithm, propose a method of making the appropriate operators of GA such as crossover and mutation probability, optimizing the weights and thresholds of BP Neural Network with the improved GA. At handwritten numeral recognition experiment, the results show that the method has faster convergence and more reliable stability, greatly improved BP neural network for learning and recognition rate.