In this paper, Kernel Fisher discriminant analysis and affine moment invariants are presented for recognizing the reference points on fabric surface. The planar images and spatial images of reference points that are indexed and attached on fabric are acquired by camera at different angle and focus, subsequently, the binary image of reference points are extracted by a series of algorithms such as filter, enhancement and binaryzation etc. Experimental result shows Kernel Fisher discriminant analysis has the better recognition ratio than routine nearest distance discriminant method. Correction recognition of reference points is the important step for the matching of multi-images and reconstruction of fabric surface morphology which provide the more information for fabric drape performance evaluation.