A multimodal feature fusion model based on non-smooth non-negative matrix factorization (nsNMF-PCNN-GRU-MSA) was proposed to address the problems of poor generalisation ability, single feature extraction mode and model uninterpretability in the fatigue driving detection methods. This model detected the level of driver fatigue by analyzing electroencephalogram (EEG) signals. A channel weighting module was designed in the shallow layer of the network, and the non-smooth non-negative matrix factorization (nsNMF) algorithm was introduced to compute the contribution of the electrode channels. A multimodal feature fusion module was designed in the middle layer of the network, where the Gramian angular field imaging method was introduced to map the 1D EEG data into a 2D image, and the spatio-temporal features of different modes were fused in parallel with the PCNN-GRU module. The multi-head self-attention (MSA) mechanism was fused in the deep layer of the network to complete the task of fatigue driving state classification. The experimental results showed that the fatigue detection accuracies of the model on the mixed samples of the SEED-VIG and SAD datasets were 93.37% and 90.78%, respectively, and the lowest accuracies for single-subject data were 86.60% and 85.59%, respectively, which were higher than those of the state-of-the-art models. The analysis method of mapping the feature activation values onto the brain topology map not only improves the interpretability of the model, but also provides a new perspective on fatigue driving detection.