Aiming at the problem of capsule defect species diversity and classification difficulty in the process of actual capsule defect detection, this paper extracts capsule defect feature based on capsule texture, shape and capsule defect region by edge detector, and then applies hierarchical SVMs multi-class classification to classifying. In order to resolve the problems of training data imbalance and the hierarchical SVM error accumulation, a algorithm of constructing hierarchical structure is proposed that takes the principle of dividing all sample data into two more imbalanced categories according to the length of training data, and then considering significant degree of capsule defect and the probability level of capsule defect occurrence. The experimental results show that compared with the method of BP neural network, the hierarchical SVMs achieved a better classification result.