In order to realize the real-time wear detection for shovel teeth during the working process of electric shovel, and prevent the mining efficiency of electric shovel from being affected by the shovel tooth wear, a shovel tooth instance segmentation model based on the improved Mask Scoring R-CNN (regional convolutional neural network) was proposed.Firstly, taking the ResNet-101 (residual network) and improved FPN (feature pyramid networks) as the backbone network, the semantic information and detail features of high and low feature layers were extracted and fused, and then the local feature layer was trimmed and normalized by combining with the ROI Align layer, so as to complete target detection and instance segmentation; then, based on the obtained shovel tooth segmentation effect image and binary mask graphic information, the pixel area of shovel tooth in the image after instance segmentation was calculated to judge its wear condition. The results showed that the mean pixel accuracy of the shovel tooth instance segmentation model with ResNet-101 and improved FPN as the backbone network was 90.76% and its mean intersection over union was 83.62% on the test set, which was 1.18% and 1.21% higher than that of the instance segmentation model with ResNet-101 and traditional FPN as the backbone network, respectively. Eight shovel tooth wear detection experiments were carried out at the electric shovel excavation site, and the fluctuation amplitude of detected wear degree of each tooth was less than 2%, and the mean square error was about 0.7, which indicated that the proposed instance segmentation model had good segmentation effect and stability for the shovel tooth, and basically met the requirements of wear detection. The research results can provide new ideas for the intelligent detection of shovel tooth wear state.