In view of the direct measurement difficulties of load swing angle of tower crane under some working conditions, obvious chattering of the system sliding mode controller and complicated adjustment of controller parameters, an adaptive sliding mode control method for tower crane based on improved fruit fly optimization algorithm was proposed. Firstly, based on the Lagrange equation, the dynamics model of the tower crane single pendulum system was obtained. Then, a linear extended state observer was designed to observe the load swing state of tower crane, and the observation results were fed back to the adaptive sliding mode controller. When constructing sliding mode surface, the hyperbolic tangent function was used instead of the common symbol function to increase its continuity and reduce chattering. Finally, the optimization strategy and search radius of the fruit fly optimization algorithm were improved, and the parameters of the adaptive sliding mode controller were optimized. The results showed that the designed linear extended state observer could track and observe the load swing angle of the tower crane with fast convergence speed and tracking error less than 1.3%. The adaptive sliding mode controller optimized by the improved fruit fly optimization algorithm not only had a good suppression effect on the load swing of tower crane, but also had strong anti-interference and robustness. The proposed control method can effectively avoid safety hazards caused by tower crane load swing while achieving precise positioning, ensuring the safety of workers and the smooth progress of the project.