A shared shallow feature network (SSFNet) was proposed to address the issues of low inference efficiency and poor segmentation performance in semantic segmentation models for UAV images. The detail branch shared the 1/4 and 1/8 stages of downsampling in the semantic branch, simplifying the downsampling process of the detail branch and improving inference efficiency. In the semantic branch, a stacked connection approach was combined with a channel decomposition mechanism to construct an efficient receptive field block (ERFB), enhancing multi-scale feature extraction with minimal additional inference cost. To integrate contextual information within the semantic branch, a fast aggregation of context (FAC) module was proposed, and a gated mechanism was utilized to supplement the semantics of the final phase during the 1/16 and 1/32 downsampling stages. During the decoding phase, a bilateral fusion module (BFM) was constructed using a hybrid activation function to fully integrate detail and semantic information. Results show that SSFNet achieves mean intersection over union scores of 68.5%, 52.7%, and 87.1% on the UAVid, LoveDA, and Potsdam datasets, respectively. SSFNet achieves an inference speed of 131.1 frames per second at a 1 024×1 024 input resolution on an NVIDIA RTX 3090 GPU, indicating strong real-time segmentation performance.