A novel neural network named multiple multiscale-feature fusion attention network (MMFA-Net) was proposed for building segmentation from high-resolution remote sensing images aiming at the disadvantages that the fully convolutional networks for building extraction have the problems of over-segmentation and internal cavity. U-Net was used as the backbone combined with multiple-extract efficient channel attention (MECA) and multiscale-feature fusion attention (MFA) structure. The MECA module was designed to strengthen the effectiveness of the feature information through the weight ratio, which was in the skip connection. The transition allocation of attention to invalid features was avoided. The multiple feature extraction was adopted to reduce the loss of effective features. The MFA module was positioned at the bottom of the model. Different spatial features and spectral dimension features were obtained through the combination of parallel continuous medium or small-scale atrous convolution and channel attention. Then the problem of pixel loss of large buildings caused by atrous convolution was alleviated. The MMFA-Net integrating the MECA and the MFA modules can promote the integrity and accuracy of building extraction results. The proposed MMFA-Net was verified on WHU, Massachusetts, and owner-drawing building datasets. MMFA-Net showed better performance compared with the other five comparison methods. The F₁-Score and IoU of MMFA-Net reached 93.33%, 87.50% at WHU datasets, 85.38%, 74.49% at Massachusetts datasets, and 88.46%, 79.31% at owner-drawing datasets, respectively.