1. College of Electrical Engineering and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China 2. Key Laboratory of Gansu Advanced Control for Industrial Processes, Lanzhou 730050, China 3. National Experimental Teaching Center of Electrical and Control Engineering, Lanzhou University of Technology, Lanzhou 730050, China
A new image super-resolution reconstruction algorithm was proposed, aiming at the problem that the image super-resolution reconstruction algorithms by using convolutional neural network as the model framework was difficult to extract multi-level feature information inside a low-resolution image, resulting in the lack of rich details in the reconstructed image. The proposed algorithm extracted shallow features from low-resolution images through initial convolution layer. Image features of different levels in a low-resolution image were obtained through a plurality of end-to-end connected multi-level continuous encoding and decoding attention residual modules, the weights of these features were generated according to different extraction difficulties, and the image features of different levels were recalibrated to obtain rich detailed features in the image. Through the up-sampling module and reconstruction convolution layer, the extracted rich detailed features and shallow features were reconstructed into high-resolution images. The comparative test results on the test sets of Set5, Set14, BSD100 and Urban100 show that the image reconstructed by the proposed algorithm is superior to the image reconstructed by a mainstream algorithm in objective evaluation index and visual effect.
Zhao-yang SONG,Xiao-qiang ZHAO,Yong-yong HUI,Hong-mei JIANG. Image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1885-1893.
Fig.1Model framework of image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding
Fig.2Comparison diagram of two encoding and decoding models
Fig.3Schematic diagram of multi-level continuous encoding and decoding model
Fig.4Schematic diagram of attention residual module structure of multi-level continuous encoding and decoding
算法
Set5
Set14
BSD100
Urban100
PSNR/dB
SSIM
PSNR/dB
SSIM
PSNR/dB
SSIM
PSNR/dB
SSIM
Bicubic
33.66
0.929 9
30.24
0.868 8
29.56
0.843 1
26.88
0.840 3
SRCNN[13]
36.66
0.954 2
32.45
0.906 7
31.36
0.887 9
29.50
0.894 6
FSRCNN[14]
37.05
0.956 0
32.66
0.909 0
31.53
0.892 0
29.88
0.902 0
VDSR[16]
37.53
0.959 0
33.05
0.913 0
31.90
0.896 0
30.77
0.914 0
DBPN[30]
38.09
0.960 0
33.85
0.919 0
32.27
0.900 0
32.55
0.932 4
RFDN[31]
38.05
0.960 6
33.68
0.918 4
32.16
0.899 4
32.12
0.927 8
A2N[22]
38.06
0.960 8
33.75
0.919 4
32.22
0.900 2
32.43
0.931 1
本研究
38.17
0.961 0
33.94
0.920 8
32.28
0.901 0
32.74
0.934 2
Tab.1Performance evaluation indicators of different image resolution algorithms on four test sets (magnification is 2)
算法
Set5
Set14
BSD100
Urban100
PSNR/dB
SSIM
PSNR/dB
SSIM
PSNR/dB
SSIM
PSNR/dB
SSIM
Bicubic
30.39
0.868 2
27.55
0.774 2
27.21
0.738 5
24.46
0.734 9
SRCNN[13]
32.75
0.909 0
29.30
0.821 5
28.41
0.786 3
26.24
0.798 9
FSRCNN[14]
33.18
0.914 0
29.37
0.824 0
28.53
0.791 0
26.43
0.808 0
VDSR[16]
33.67
0.921 0
29.78
0.832 0
28.83
0.799 0
27.14
0.829 0
RFDN[31]
34.41
0.927 3
30.34
0.842 0
29.09
0.805 0
28.21
0.852 5
A2N[22]
34.47
0.927 9
30.44
0.843 7
29.14
0.805 9
28.41
0.857 0
本研究
34.58
0.928 8
30.51
0.845 6
29.20
0.807 9
28.66
0.863 4
Tab.2Performance evaluation indicators of different image resolution algorithms on four test sets (magnification is 3)
算法
Set5
Set14
BSD100
Urban100
PSNR/dB
SSIM
PSNR/dB
SSIM
PSNR/dB
SSIM
PSNR/dB
SSIM
Bicubic
28.42
0.810 4
26.00
0.702 7
25.96
0.667 5
23.14
0.657 7
SRCNN[13]
30.48
0.862 8
27.50
0.751 3
26.90
0.710 1
24.52
0.722 1
FSRCNN[14]
30.72
0.866 0
27.61
0.755 0
26.98
0.715 0
24.62
0.728 0
VDSR[16]
31.35
0.883 0
28.02
0.768 0
27.29
0.725 1
25.18
0.754 0
DBPN[30]
32.47
0.898 0
28.82
0.786 0
27.72
0.740 0
26.38
0.794 6
RFDN[31]
32.24
0.895 2
28.61
0.781 9
27.57
0.736 0
26.11
0.785 8
A2N[22]
32.30
0.896 6
28.71
0.784 2
27.61
0.737 4
26.27
0.792 0
本研究
32.37
0.897 1
28.76
0.785 7
27.55
0.734 0
26.41
0.797 4
Tab.3Performance evaluation indicators of different image resolution algorithms on four test sets (magnification is 4)
Fig.5Comparison images of img004 reconstructed by different image resolution algorithms (magnification is 4)
Fig.6Comparison images of img061 reconstructed by different image resolution algorithms (magnification is 4)
Fig.7Comparison images of img096 reconstructed by different image resolution algorithms (magnification is 4)
网络
PSNR/dB
未组合ED1、ED2、ED3、Attention
31.57
ED1
31.64
ED1+ED2
32.01
ED1+ED2+ED3
32.16
ED1+ED2+ED3+Attention
32.37
Tab.4Ablation study results of attention residual module based on multi-level continuous encoding and decoding (magnification is 4)
Fig.8Performance comparison of different image resolution algorithms on Urban100 test set (magnification is 4)
[20]
ZHANG Y, TIAN Y, KONG Y, et al. Residual dense network for image super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2472-2481.
[21]
ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks [C]// European Conference on Computer Vision. [S.l.]: Springer, 2018: 294-301.
[22]
CHEN H, GU J, ZHANG Z. Attention in attention network for image super-resolution [EB/OL]. [2022-04-21]. https://arxiv.org/pdf/2104.09497.pdf.
[23]
AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 126-135.
[24]
BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding [C]// Proceedings British Machine Vision Conference. Surrey: [s.n.], 2012: 1-10.
[25]
ZEYDE R, ELAD M, PROTTER M, et al. On single image scale-up using sparse-representations [C]// International Conference on Curves and Surfaces. [S.l.]: Springer, 2010: 711-730.
[26]
MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics [C]// Proceedings of the IEEE International Conference on Computer Vision. Vancouver: IEEE, 2001: 416-423.
[27]
HUANG J, SINGH A, AHUJA N, et al. Single image super-resolution from transformed self-exemplars [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 5197-5206
[28]
KINGMA D P, BA J L. ADAM: a method for stochastic optimization [EB/OL]. [2022-04-21]. https://arxiv.org/pdf/1412.6980.pdf.
[29]
WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612
doi: 10.1109/TIP.2003.819861
[30]
HARIS M, SHAKHNAROVICH G, UKITA N Deep back-projection networks for super-resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (12): 4323- 4337
doi: 10.1109/TPAMI.2020.3002836
[31]
LIU J, TANG J, WU G. Residual feature distillation network for lightweight image super-resolution [C]// European Conference on Computer Vision. [S.l.]: Springer, 2020: 41-55.
[1]
CHEN H, HE X, QING L, et al Real-world single image super-resolution: a brief review[J]. Information Fusion, 2022, 79: 124- 145
doi: 10.1016/j.inffus.2021.09.005
[2]
REBECQ H, RANFTL R, KOLTUN V, et al. Events-to-video: bringing modern computer vision to event cameras [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 3857-3866.
[3]
WANG Z, JIANG K, YI P, et al Ultra-dense GAN for satellite imagery super-resolution[J]. Neurocomputing, 2020, 398: 328- 373
doi: 10.1016/j.neucom.2019.03.106
[4]
LIU H, GU Y, WANG T, et al Satellite video super-resolution based on adaptively spatiotemporal neighbors and nonlocal similarity regularization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58 (12): 8372- 8383
doi: 10.1109/TGRS.2020.2987400
[5]
YUE L, SHEN H, LI J, et al Image super-resolution: the techniques, applications, and future[J]. Signal Processing, 2016, 128: 389- 408
doi: 10.1016/j.sigpro.2016.05.002
[6]
DENG L, YU D. Deep learning: methods and applications [M]. [S.l.]: Now Foundations and Trends, 2014: 197-387.
[7]
JORDAN M I, MITCHELL T M Machine learning: trends, perspectives, and prospects[J]. Science, 2015, 349 (6245): 255- 260
doi: 10.1126/science.aaa8415
[8]
LECUN Y, BENGIO Y, HINTON G Deep learning[J]. Nature, 2015, 521: 436- 444
doi: 10.1038/nature14539
[9]
ANWAR S, KHAN S, BARNES N A deep journey into super-resolution: a survey[J]. ACM Computing Surveys, 2020, 53 (3): 1- 34
[10]
WANG Z, CHEN J, HOI S C H Deep learning for image super-resolution: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (10): 3365- 3387
doi: 10.1109/TPAMI.2020.2982166
[11]
SONG Z, ZHAO X, JIANG H Gradual deep residual network for super-resolution[J]. Multimedia Tools and Applications, 2021, 80: 9765- 9778
doi: 10.1007/s11042-020-10152-9
[12]
SONG Z, ZHAO X, HUI Y, et al Progressive back-projection network for COVID-CT super-resolution[J]. Computer Methods and Programs in Biomedicine, 2021, 208: 106193
doi: 10.1016/j.cmpb.2021.106193
[13]
DONG C, LOY C C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (2): 295- 307
[14]
DONG C, LOY C C, TANG X, et al. Accelerating the super-resolution convolutional neural network [C]// European Conference on Computer Vision. [S.l.]: Springer, 2016: 391-407.
[15]
SHI W, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
[16]
KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646-1654.
[17]
KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1637-1645.
[18]
LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 136-144.
