An efficient lightweight model, U-SAM, was proposed aiming at the requirement for lightweight, high-precision and high-speed performance in real-time semantic segmentation task on edge device. A multi-scale dilated convolution fusion module was designed in the encoder stage based on an improved U-Net architecture in order to enhance feature representation capability and significantly reduce computational complexity. Channel attention and pixel attention mechanism were introduced in the decoder stage in order to achieve effective modeling of key feature information and cross-scale fusion. Self-attention mechanism and convolution operation were integrated in the bottleneck section in order to efficiently capture long-range dependency and improve the generalization capability of the model. Experiments on the small-scale CamVid dataset demonstrated that U-SAM required only about 1.3×10⁶ parameters in order to achieve segmentation accuracy of 70.2% mIoU, with inference speed reaching 69 frame/s. The overall performance of U-SAM is significantly superior to several mainstream lightweight real-time semantic segmentation models, providing a high-performance solution for real-time semantic segmentation application in resource-constrained scenarios such as autonomous driving, intelligent surveillance and remote sensing satellite.