Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (9): 1885-1893    DOI: 10.3785/j.issn.1008-973X.2023.09.020
    
Image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding
Zhao-yang SONG1,2,3(),Xiao-qiang ZHAO1,2,3,*(),Yong-yong HUI1,2,3,Hong-mei JIANG1,2,3
1. College of Electrical Engineering and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China
2. Key Laboratory of Gansu Advanced Control for Industrial Processes, Lanzhou 730050, China
3. National Experimental Teaching Center of Electrical and Control Engineering, Lanzhou University of Technology, Lanzhou 730050, China
Download: HTML     PDF(2659KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A new image super-resolution reconstruction algorithm was proposed, aiming at the problem that the image super-resolution reconstruction algorithms by using convolutional neural network as the model framework was difficult to extract multi-level feature information inside a low-resolution image, resulting in the lack of rich details in the reconstructed image. The proposed algorithm extracted shallow features from low-resolution images through initial convolution layer. Image features of different levels in a low-resolution image were obtained through a plurality of end-to-end connected multi-level continuous encoding and decoding attention residual modules, the weights of these features were generated according to different extraction difficulties, and the image features of different levels were recalibrated to obtain rich detailed features in the image. Through the up-sampling module and reconstruction convolution layer, the extracted rich detailed features and shallow features were reconstructed into high-resolution images. The comparative test results on the test sets of Set5, Set14, BSD100 and Urban100 show that the image reconstructed by the proposed algorithm is superior to the image reconstructed by a mainstream algorithm in objective evaluation index and visual effect.



Key wordssuper-resolution reconstruction      convolutional neural network      multi-level continuous encoding and decoding      attention      multi-level feature information     
Received: 24 May 2022      Published: 16 October 2023
CLC:  TP 391  
Fund:  国家重点研发计划资助项目(2020YFB1713600);国家自然科学基金资助项目(61763029);甘肃省科技计划资助项目(21YF5GA072, 21JR7RA206);甘肃省教育厅产业支撑计划资助项目(2021CYZC-02)
Corresponding Authors: Xiao-qiang ZHAO     E-mail: szy@lut.edu.cn;xqzhao@lut.edu.cn
Cite this article:

Zhao-yang SONG,Xiao-qiang ZHAO,Yong-yong HUI,Hong-mei JIANG. Image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1885-1893.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.09.020     OR     https://www.zjujournals.com/eng/Y2023/V57/I9/1885


基于多级连续编码与解码的图像超分辨率重建算法

以卷积神经网络为模型框架的图像超分辨率重建算法难以提取低分辨率图像内部的多层次特征信息, 导致重建图像缺少丰富细节, 为此提出新的图像超分辨率重建算法. 所提算法通过初始卷积层从低分辨率图像提取浅层特征; 通过多个端对端连接的多级连续编码与解码的注意力残差模块获取低分辨率图像内部不同层级的图像特征, 按照不同的提取难度生成这些特征的权重, 重新校准不同层次的图像特征, 获取图像内部丰富的细节特征;通过上采样模块和重建卷积层将提取到的丰富细节特征和浅层特征重建成高分辨率图像. 在Set5、Set14、BSD100和Urban100测试集上进行的对比测试结果表明,使用所提算法重建的图像在客观评价指标、视觉效果上均优于使用主流算法重建的图像.


关键词: 超分辨率重建,  卷积神经网络,  多级连续编码与解码,  注意力,  多层次特征信息 
Fig.1 Model framework of image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding
Fig.2 Comparison diagram of two encoding and decoding models
Fig.3 Schematic diagram of multi-level continuous encoding and decoding model
Fig.4 Schematic diagram of attention residual module structure of multi-level continuous encoding and decoding
算法 Set5 Set14 BSD100 Urban100
PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM
Bicubic 33.66 0.929 9 30.24 0.868 8 29.56 0.843 1 26.88 0.840 3
SRCNN[13] 36.66 0.954 2 32.45 0.906 7 31.36 0.887 9 29.50 0.894 6
FSRCNN[14] 37.05 0.956 0 32.66 0.909 0 31.53 0.892 0 29.88 0.902 0
VDSR[16] 37.53 0.959 0 33.05 0.913 0 31.90 0.896 0 30.77 0.914 0
DBPN[30] 38.09 0.960 0 33.85 0.919 0 32.27 0.900 0 32.55 0.932 4
RFDN[31] 38.05 0.960 6 33.68 0.918 4 32.16 0.899 4 32.12 0.927 8
A2N[22] 38.06 0.960 8 33.75 0.919 4 32.22 0.900 2 32.43 0.931 1
本研究 38.17 0.961 0 33.94 0.920 8 32.28 0.901 0 32.74 0.934 2
Tab.1 Performance evaluation indicators of different image resolution algorithms on four test sets (magnification is 2)
算法 Set5 Set14 BSD100 Urban100
PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM
Bicubic 30.39 0.868 2 27.55 0.774 2 27.21 0.738 5 24.46 0.734 9
SRCNN[13] 32.75 0.909 0 29.30 0.821 5 28.41 0.786 3 26.24 0.798 9
FSRCNN[14] 33.18 0.914 0 29.37 0.824 0 28.53 0.791 0 26.43 0.808 0
VDSR[16] 33.67 0.921 0 29.78 0.832 0 28.83 0.799 0 27.14 0.829 0
RFDN[31] 34.41 0.927 3 30.34 0.842 0 29.09 0.805 0 28.21 0.852 5
A2N[22] 34.47 0.927 9 30.44 0.843 7 29.14 0.805 9 28.41 0.857 0
本研究 34.58 0.928 8 30.51 0.845 6 29.20 0.807 9 28.66 0.863 4
Tab.2 Performance evaluation indicators of different image resolution algorithms on four test sets (magnification is 3)
算法 Set5 Set14 BSD100 Urban100
PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM
Bicubic 28.42 0.810 4 26.00 0.702 7 25.96 0.667 5 23.14 0.657 7
SRCNN[13] 30.48 0.862 8 27.50 0.751 3 26.90 0.710 1 24.52 0.722 1
FSRCNN[14] 30.72 0.866 0 27.61 0.755 0 26.98 0.715 0 24.62 0.728 0
VDSR[16] 31.35 0.883 0 28.02 0.768 0 27.29 0.725 1 25.18 0.754 0
DBPN[30] 32.47 0.898 0 28.82 0.786 0 27.72 0.740 0 26.38 0.794 6
RFDN[31] 32.24 0.895 2 28.61 0.781 9 27.57 0.736 0 26.11 0.785 8
A2N[22] 32.30 0.896 6 28.71 0.784 2 27.61 0.737 4 26.27 0.792 0
本研究 32.37 0.897 1 28.76 0.785 7 27.55 0.734 0 26.41 0.797 4
Tab.3 Performance evaluation indicators of different image resolution algorithms on four test sets (magnification is 4)
Fig.5 Comparison images of img004 reconstructed by different image resolution algorithms (magnification is 4)
Fig.6 Comparison images of img061 reconstructed by different image resolution algorithms (magnification is 4)
Fig.7 Comparison images of img096 reconstructed by different image resolution algorithms (magnification is 4)
网络 PSNR/dB
未组合ED1、ED2、ED3、Attention 31.57
ED1 31.64
ED1+ED2 32.01
ED1+ED2+ED3 32.16
ED1+ED2+ED3+Attention 32.37
Tab.4 Ablation study results of attention residual module based on multi-level continuous encoding and decoding (magnification is 4)
Fig.8 Performance comparison of different image resolution algorithms on Urban100 test set (magnification is 4)
[20]   ZHANG Y, TIAN Y, KONG Y, et al. Residual dense network for image super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2472-2481.
[21]   ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks [C]// European Conference on Computer Vision. [S.l.]: Springer, 2018: 294-301.
[22]   CHEN H, GU J, ZHANG Z. Attention in attention network for image super-resolution [EB/OL]. [2022-04-21]. https://arxiv.org/pdf/2104.09497.pdf.
[23]   AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 126-135.
[24]   BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding [C]// Proceedings British Machine Vision Conference. Surrey: [s.n.], 2012: 1-10.
[25]   ZEYDE R, ELAD M, PROTTER M, et al. On single image scale-up using sparse-representations [C]// International Conference on Curves and Surfaces. [S.l.]: Springer, 2010: 711-730.
[26]   MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics [C]// Proceedings of the IEEE International Conference on Computer Vision. Vancouver: IEEE, 2001: 416-423.
[27]   HUANG J, SINGH A, AHUJA N, et al. Single image super-resolution from transformed self-exemplars [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 5197-5206
[28]   KINGMA D P, BA J L. ADAM: a method for stochastic optimization [EB/OL]. [2022-04-21]. https://arxiv.org/pdf/1412.6980.pdf.
[29]   WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612
doi: 10.1109/TIP.2003.819861
[30]   HARIS M, SHAKHNAROVICH G, UKITA N Deep back-projection networks for super-resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (12): 4323- 4337
doi: 10.1109/TPAMI.2020.3002836
[31]   LIU J, TANG J, WU G. Residual feature distillation network for lightweight image super-resolution [C]// European Conference on Computer Vision. [S.l.]: Springer, 2020: 41-55.
[1]   CHEN H, HE X, QING L, et al Real-world single image super-resolution: a brief review[J]. Information Fusion, 2022, 79: 124- 145
doi: 10.1016/j.inffus.2021.09.005
[2]   REBECQ H, RANFTL R, KOLTUN V, et al. Events-to-video: bringing modern computer vision to event cameras [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 3857-3866.
[3]   WANG Z, JIANG K, YI P, et al Ultra-dense GAN for satellite imagery super-resolution[J]. Neurocomputing, 2020, 398: 328- 373
doi: 10.1016/j.neucom.2019.03.106
[4]   LIU H, GU Y, WANG T, et al Satellite video super-resolution based on adaptively spatiotemporal neighbors and nonlocal similarity regularization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58 (12): 8372- 8383
doi: 10.1109/TGRS.2020.2987400
[5]   YUE L, SHEN H, LI J, et al Image super-resolution: the techniques, applications, and future[J]. Signal Processing, 2016, 128: 389- 408
doi: 10.1016/j.sigpro.2016.05.002
[6]   DENG L, YU D. Deep learning: methods and applications [M]. [S.l.]: Now Foundations and Trends, 2014: 197-387.
[7]   JORDAN M I, MITCHELL T M Machine learning: trends, perspectives, and prospects[J]. Science, 2015, 349 (6245): 255- 260
doi: 10.1126/science.aaa8415
[8]   LECUN Y, BENGIO Y, HINTON G Deep learning[J]. Nature, 2015, 521: 436- 444
doi: 10.1038/nature14539
[9]   ANWAR S, KHAN S, BARNES N A deep journey into super-resolution: a survey[J]. ACM Computing Surveys, 2020, 53 (3): 1- 34
[10]   WANG Z, CHEN J, HOI S C H Deep learning for image super-resolution: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (10): 3365- 3387
doi: 10.1109/TPAMI.2020.2982166
[11]   SONG Z, ZHAO X, JIANG H Gradual deep residual network for super-resolution[J]. Multimedia Tools and Applications, 2021, 80: 9765- 9778
doi: 10.1007/s11042-020-10152-9
[12]   SONG Z, ZHAO X, HUI Y, et al Progressive back-projection network for COVID-CT super-resolution[J]. Computer Methods and Programs in Biomedicine, 2021, 208: 106193
doi: 10.1016/j.cmpb.2021.106193
[13]   DONG C, LOY C C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (2): 295- 307
[14]   DONG C, LOY C C, TANG X, et al. Accelerating the super-resolution convolutional neural network [C]// European Conference on Computer Vision. [S.l.]: Springer, 2016: 391-407.
[15]   SHI W, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
[16]   KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646-1654.
[17]   KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1637-1645.
[18]   LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 136-144.
[1] Hai-feng LI,Xue-ying ZHANG,Shu-fei DUAN,Hai-rong JIA,Hui-zhi LIANG. Fusing generative adversarial network and temporal convolutional network for Mandarin emotion recognition[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1865-1875.
[2] Xiao-qiang ZHAO,Ze WANG,Zhao-yang SONG,Hong-mei JIANG. Image super-resolution reconstruction based on dynamic attention network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1487-1494.
[3] Hui-xin WANG,Xiang-rong TONG. Research progress of recommendation system based on knowledge graph[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1527-1540.
[4] Dian-hai WANG,Rui XIE,Zheng-yi CAI. Prediction of urban interrupted traffic flow based on optimal convergence time interval[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1607-1617.
[5] Xiu-lan SONG,Zhao-hang DONG,Hang-guan SHAN,Wei-jie LU. Vehicle trajectory prediction based on temporal-spatial multi-head attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(8): 1636-1643.
[6] Xiao-yan LI,Peng WANG,Jia GUO,Xue LI,Meng-yu SUN. Multi branch Siamese network target tracking based on double attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1307-1316.
[7] Yun-hong LI,Jiao-jiao DUAN,Xue-ping SU,Lei-tao ZHANG,Hui-kang YU,Xing-rui LIU. Calligraphy generation algorithm based on improved generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1326-1334.
[8] Chun-juan LIU,Ze QIAO,Hao-wen YAN,Xiao-suo WU,Jia-wei WANG,Yu-qiang XIN. Semantic segmentation network for remote sensing image based on multi-scale mutual attention[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1335-1344.
[9] Wei QUAN,Yong-qing CAI,Chao WANG,Jia SONG,Hong-kai SUN,Lin-xuan LI. VR sickness estimation model based on 3D-ResNet two-stream network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1345-1353.
[10] Yan ZHANG,Jing-xue SUN,Ye-mei SUN,Shu-dong LIU,Chuan-qi WANG. Lightweight object detection based on split attention and linear transformation[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(6): 1195-1204.
[11] Hai-bo ZHANG,Lei CAI,Jun-ping REN,Ru-yan WANG,Fu LIU. Efficient and adaptive semantic segmentation network based on Transformer[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(6): 1205-1214.
[12] Jun HAN,Xiao-ping YUAN,Zhun WANG,Ye CHEN. UAV dense small target detection algorithm based on YOLOv5s[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(6): 1224-1233.
[13] Xin-dong LV,Jiao LI,Zhen-nan DENG,Hao FENG,Xin-tong CUI,Hong-xia DENG. Structured image super-resolution network based on improved Transformer[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 865-874.
[14] Xue-yong XIANG,Li WANG,Wen-peng ZONG,Guang-yun LI. Point cloud instance segmentation based on attention mechanism KNN and ASIS module[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 875-882.
[15] Jian-zhao ZHANG,Ji-chang GUO,Yu-dong WANG. Underwater image enhancement algorithm via fusing reverse medium transmission map[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 921-929.