An end-to-end underwater image enhancement algorithm via fusing reverse medium transmission map was proposed to resolve the problem of quality degradation in the underwater images. The original RGB map and the reverse medium transmission map obtained by the traditional method were fed into two encoders of the proposed two-stream convolutional neural network. The information of two images was fully integrated through the cross-modality feature fusion module, and the network could learn the characteristics of underwater optical images well. The feature expression capabilities could be further strengthened through the feature enhancement modules. The decoder and encoder were connected via a residual decoding module to supply the RGB features. The underwater image enhancement algorithm via fusing reverse medium transmission map processes information from coarse to fine through cross-modality and cross-scale information fusion, and finally outputs an enhanced RGB image. The experimental results showed that the proposed algorithm could effectively improve the visual quality of underwater images. Taking into account both subjective evaluation and objective evaluation, the proposed algorithm was better than the six competing algorithms of white balance WB and histogram equalization HE, Water-Net, UGAN, UWCNN, Ucolor.