A multi-scale defect segmentation model, EL-SegFormer, was proposed based on an improved SegFormer architecture, aiming at the defects affecting the lifetime and efficiency in solar cell manufacturing. The model was specifically designed to segment defects in solar cells, providing a reliable detection tool for manufacturers. A lightweight modulation module was incorporated in the shallow layers of the network, and multi-head hybrid convolutions were used to capture defect features across various scales. Fixed-scale convolutions and receptive fields were employed to effectively capture early local information in the network. Diverse defects in solar cells can be accurately located by aggregating the extracted features. A hierarchical encoder structure was employed to integrate multi-scale contextual information from shallow to deep layers into the decoder. The decoder utilized a lightweight multi-layer perceptron to consolidate feature information from different levels and generate segmentation masks. The model was loaded and traversed to compute the mean intersection over union (MIoU) using the defect image segmentation masks and label masks. Experimental results indicated that EL-SegFormer, with only 68.2 M parameters, achieved the MIoU of 67.60% on the Buerhop2018 dataset, surpassing recent state-of-the-art models. This outstanding performance indicates the model’s strong potential for addressing complex solar cell defect segmentation tasks, opening up promising avenues for its application in the solar cell manufacturing industry.