Abstract: To study 16 kinds of common minerals, the authors collected 1000 images for each type, and then divided them into training set, validation set and test set. Before putting the images into the model, the authors selected a random area of each image for data augmentation. After training the InceptionV3 model with 70000 steps, the authors obtained an 81% accuracy in the test set. Through improving the loss function and introducing the Center Loss, the authors raised the accuracy to 86% after training 400000 steps. The obfuscation matrix shows that, the recognition accuracies for the minerals with obvious appearance characteristics such as malachite are higher while those for other minerals like sphalerite are less due to the obfuscation with other minerals. The analysis of the feature map shows that the model extracts the radial feature of malachite perfectly, and the feature vector of mineral image aggregate is in a high degree, which also can prove the reliability of the model.