On the Brittleness of Handwritten Digit Recognition Models

Handwritten digit recognition is an important benchmark task in computer vision. Learning algorithms and feature representations which offer excellent performance for this task have been known for some time. Here, we focus on two major practical considerations: the relationship between the the amount of training data and error rate (corresponding to the effort to collect training data to build a model with a given maximum error rate) and the transferability of models' expertise between different datasets (corresponding to the usefulness for general handwritten digit recognition). While the relationship between amount of training data and error rate is very stable and to some extent independent of the specific dataset used—only the classifier and feature representation have significant effect—it has proven to be impossible to transfer low error rates on one or two pooled datasets to similarly low error rates on another dataset. We have called this weakness brittleness, inspired by an old Artificial Intelligence term that means the same thing. This weakness may be a general weakness of trained image classification systems.

Download Full-text

Handwritten Digit Recognition Using K Nearest-Neighbor, Radial-Basis Function, and Backpropagation Neural Networks

Neural Computation ◽

10.1162/neco.1991.3.3.440 ◽

1991 ◽

Vol 3 (3) ◽

pp. 440-449 ◽

Cited By ~ 112

Author(s):

Yuchun Lee

Keyword(s):

Nearest Neighbor ◽

Error Rates ◽

K Nearest Neighbor ◽

Handwritten Digit Recognition ◽

Training Time ◽

Digit Recognition ◽

Knn Classifier ◽

Backpropagation Network ◽

Handwritten Digit ◽

Classification Time

Results of recent research suggest that carefully designed multilayer neural networks with local “receptive fields” and shared weights may be unique in providing low error rates on handwritten digit recognition tasks. This study, however, demonstrates that these networks, radial basis function (RBF) networks, and k nearest-neighbor (kNN) classifiers, all provide similar low error rates on a large handwritten digit database. The backpropagation network is overall superior in memory usage and classification time but can provide “false positive” classifications when the input is not a digit. The backpropagation network also has the longest training time. The RBF classifier requires more memory and more classification time, but less training time. When high accuracy is warranted, the RBF classifier can generate a more effective confidence judgment for rejecting ambiguous inputs. The simple kNN classifier can also perform handwritten digit recognition, but requires a prohibitively large amount of memory and is much slower at classification. Nevertheless, the simplicity of the algorithm and fast training characteristics makes the kNN classifier an attractive candidate in hardware-assisted classification tasks. These results on a large, high input dimensional problem demonstrate that practical constraints including training time, memory usage, and classification time often constrain classifier selection more strongly than small differences in overall error rate.

Download Full-text

Deep, Big, Simple Neural Nets for Handwritten Digit Recognition

Neural Computation ◽

10.1162/neco_a_00052 ◽

2010 ◽

Vol 22 (12) ◽

pp. 3207-3220 ◽

Cited By ~ 427

Author(s):

Dan Claudiu Cireşan ◽

Ueli Meier ◽

Luca Maria Gambardella ◽

Jürgen Schmidhuber

Keyword(s):

Error Rate ◽

Neural Nets ◽

Multilayer Perceptrons ◽

Handwritten Digit Recognition ◽

Digit Recognition ◽

Training Images ◽

Speed Up ◽

Handwritten Digit

Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35% error rate on the MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images to avoid overfitting, and graphics cards to greatly speed up learning.

Download Full-text