An end-to-end surface defects detection and recognition method was proposed to solve the problem of high-precision detection of aluminum strip surface defects and the poor recognition rate of traditional algorithms. The average image was quickly calculated from the initial image sequence of aluminum strip surface, which was regarded as defect-free background image and was used to initialize the background model of the ViBe algorithm. The ViBe algorithm was used to segment the defect region from the current image. Median filtering and morphological operation were performed on the binary image of defect region to remove noise points and repair edges in order to accurately extract the defect region. The current image was used to update the ViBe background model in real time in order to increase the adaptability of the algorithm to illumination changes. The image of external rectangular region of the defect was extracted, normalized, and input into the trained convolutional neural networks for recognition and classification. The classification result was obtained. The experimental results show that the proposed method has a defect detection rate of 93.02% and a defect recognition rate of 99.86%, which has good application value.