Existing methods show insufficient capability in simultaneously capturing global semantic information and local detailed features. As a result, missed detections and false detections of building information often occur in complex scenes. To address this problem, a dual-branch feature fusion and mutual enhancement network (DFFME-Net) based on BuildFormer was proposed. Firstly, VGG13 was employed as the backbone to extract local features, and a multi-scale global-local feature fusion method (MGLFF) was designed to fuse features from both branches at each stage, generating enriched feature representations. Secondly, to break the independence between the two branches and promote mutual reinforcement in feature extraction, a dual-branch feature fusion module (DFM) was introduced. Finally, to enhance the network’s focus on regions of interest, a channel prior convolutional attention mechanism (CPCA) was incorporated into the DFM, enabling improved feature representation and dynamically allocating attention weights. Experiments were conducted on the WHU and Massachusetts building datasets to verify the effectiveness and applicability of the proposed network. DFFME-Net achieved an IoU of 91.81% and an F1-score of 95.73% on the WHU test set, and 77.01% and 87.01% on the Massachusetts test set, respectively, outperforming several advanced models. Results indicate that the proposed network has strong practical application value in high-resolution remote sensing building extraction tasks.