Difference between non-liveness attack types is neglected, and adverse impact of category imbalance between liveness and non-liveness samples on model training is not considered in existing studies of silent liveness detection. In this paper, non-liveness attacks were subdivided into two categories, print attack and display attack, which transformed silent liveness detection from traditional two-classification problem into multi-classification problem. And the cross-entropy was used as the loss function to train network model. Thus, the disadvantage of binary classification and category imbalance can be eliminated, common features of the non-liveness face samples were likely to be identified more accurately through model training, and the accuracy of the network model was improved for non-liveness recognition. Moreover, a two-stream feature fusion the network model was constructed to further improve the feature representation capacity of the network model, which adopted the attention mechanism to adaptively fuse the feature vectors extracted from RGB and YCrCb. Abundant comparative experiments were performed on four public datasets, CASIA-FASD, Replay-Attack, MSU-MFSD and OULU-NPU. Experimental results indicate that silent liveness detection model adopting multi-classification strategy and feature fusion can effectively reduce the classification error and improve over-generalization ability.