A novel signal analysis method combining variational mode decomposition (VMD), particle swarm optimization (PSO), and random forest (RF) was proposed aiming at the signal processing and feature recognition problems in the vibro-acoustic detection for arc magnet internal defects. A fitness function representing the processing performance of VMD is constructed by both the mode energies and the center frequency difference of adjacent modes, in which two parameters of VMD, including the decomposition number and the penalty factor, are used as the function variables. The parameter optimization of VMD is performed by PSO, which is responsible for searching for the minimum value of the function in the VMD parameter space, and the parameters corresponding to the found minimum value can be regarded as the optimal parameter setting of VMD. The obtained parameters are used to achieve the optimal VMD decomposition of the signal, and the characteristic mode is determined by calculating the energy of modes. The zero-crossing rate, the spectral centroid, and the maximum peak frequency are extracted from the selected mode to jointly reflect the characteristic information of the internal defects of arc magnets. RF classifier is utilized to identify the extracted features to judge the existence of internal defects. Experimental results show that the proposed method can realize accurate and efficient internal defect detection for different types of arc magnets.