Existing cross-modal semantic segmentation networks ignore the complementarity among cross-modal features and the correlation among features of different scales, resulting in blurred boundaries and poor segmentation accuracy for distant small targets under complex lighting conditions. A new cross-modal semantic segmentation algorithm (EMFFNet) was proposed. In the encoder, based on the different semantic characteristics of shallow and deep layers, a multi-scale attention fusion module and a cross-modal feature weighted fusion module were designed to capture detailed texture information and high-order semantic information, respectively. A channel enhancement module and edge supervision strategy were introduced into the decoder to assist training, and the target boundary perception ability was strengthened. Compared with the mainstream algorithms, EMFFNet achieves the best evaluation metrics on the MFNet urban street view dataset, with mean accuracy and mean intersection-over-union of 76.1% and 59.4%, respectively. Experimental results show that EMFFNet effectively alleviates edge blurring and enhances segmentation accuracy of distant small targets under varying illumination.