A method was proposed to directly establish the mapping relationship between point clouds and panoramic images in response to the complex calibration process of sensor external parameters when obtaining color point clouds using the ground 3D scanner. Firstly, an improved Zernike moment sub-pixel edge extraction algorithm based on one-dimensional maximum entropy was proposed to locate the target sphere from the panoramic image, and the target sphere was extracted from the point cloud according to the 3D geometric characteristics. Then, the extraction result was used as the registration primitive, and the primitive triangle was constructed in the spatial spherical coordinate. The primitive pairing was completed by the minimum angular distance difference method, and the initial mapping relationship between the point cloud and the panoramic image was established. Finally, aiming at the mapping deviation caused by local image distortion, a hybrid algorithm based on improved Levenberg-Marquardt and free-form deformation combination was proposed to optimize the mapping relationship between data pixel by pixel. The feasibility of the proposed method is verified by the experimental data of multiple scenarios. The results show that the extraction rate of the target from the point cloud and the image is high, and the target recognized by the point cloud and the image is successfully paired by the minimum angle difference method. Compared with the traditional Zernike moment extraction target, the initial mapping error was reduced by 61.1% for the improved Zernike moment. After the optimization of the hybrid algorithm, the mapping error between the point cloud and the panoramic image was about 1 pixel, and the data mapping result was stable and not affected by the position of the station and the point cloud density.