Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (9): 1885-1893    DOI: 10.3785/j.issn.1008-973X.2023.09.020
电子、通信与自动控制技术     
基于多级连续编码与解码的图像超分辨率重建算法
宋昭漾1,2,3(),赵小强1,2,3,*(),惠永永1,2,3,蒋红梅1,2,3
1. 兰州理工大学 电气工程与信息工程学院，甘肃 兰州 730050
2. 甘肃省工业过程先进控制重点实验室，甘肃 兰州 730050
3. 兰州理工大学 国家级电气与控制工程实验教学中心，甘肃 兰州 730050
Image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding
Zhao-yang SONG1,2,3(),Xiao-qiang ZHAO1,2,3,*(),Yong-yong HUI1,2,3,Hong-mei JIANG1,2,3
1. College of Electrical Engineering and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China
2. Key Laboratory of Gansu Advanced Control for Industrial Processes, Lanzhou 730050, China
3. National Experimental Teaching Center of Electrical and Control Engineering, Lanzhou University of Technology, Lanzhou 730050, China
 全文: PDF(2659 KB)   HTML
摘要：

以卷积神经网络为模型框架的图像超分辨率重建算法难以提取低分辨率图像内部的多层次特征信息, 导致重建图像缺少丰富细节, 为此提出新的图像超分辨率重建算法. 所提算法通过初始卷积层从低分辨率图像提取浅层特征； 通过多个端对端连接的多级连续编码与解码的注意力残差模块获取低分辨率图像内部不同层级的图像特征, 按照不同的提取难度生成这些特征的权重, 重新校准不同层次的图像特征, 获取图像内部丰富的细节特征；通过上采样模块和重建卷积层将提取到的丰富细节特征和浅层特征重建成高分辨率图像. 在Set5、Set14、BSD100和Urban100测试集上进行的对比测试结果表明，使用所提算法重建的图像在客观评价指标、视觉效果上均优于使用主流算法重建的图像.
关键词： 超分辨率重建;  卷积神经网络;  多级连续编码与解码;  注意力;  多层次特征信息    
Abstract:

A new image super-resolution reconstruction algorithm was proposed, aiming at the problem that the image super-resolution reconstruction algorithms by using convolutional neural network as the model framework was difficult to extract multi-level feature information inside a low-resolution image, resulting in the lack of rich details in the reconstructed image. The proposed algorithm extracted shallow features from low-resolution images through initial convolution layer. Image features of different levels in a low-resolution image were obtained through a plurality of end-to-end connected multi-level continuous encoding and decoding attention residual modules, the weights of these features were generated according to different extraction difficulties, and the image features of different levels were recalibrated to obtain rich detailed features in the image. Through the up-sampling module and reconstruction convolution layer, the extracted rich detailed features and shallow features were reconstructed into high-resolution images. The comparative test results on the test sets of Set5, Set14, BSD100 and Urban100 show that the image reconstructed by the proposed algorithm is superior to the image reconstructed by a mainstream algorithm in objective evaluation index and visual effect.
Key words: super-resolution reconstruction    convolutional neural network    multi-level continuous encoding and decoding    attention    multi-level feature information
收稿日期: 2022-05-24 出版日期: 2023-10-16
CLC:  TP 391  
基金资助: 国家重点研发计划资助项目(2020YFB1713600)；国家自然科学基金资助项目(61763029)；甘肃省科技计划资助项目(21YF5GA072, 21JR7RA206)；甘肃省教育厅产业支撑计划资助项目(2021CYZC-02)
通讯作者: 赵小强     E-mail: szy@lut.edu.cn;xqzhao@lut.edu.cn
作者简介: 宋昭漾（1995—），男，博士生，从事图像超分辨率研究. orcid.org/0000-0003-0754-0846. E-mail： szy@lut.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
宋昭漾
赵小强
惠永永
蒋红梅

引用本文:

宋昭漾,赵小强,惠永永,蒋红梅. 基于多级连续编码与解码的图像超分辨率重建算法[J]. 浙江大学学报(工学版), 2023, 57(9): 1885-1893.

Zhao-yang SONG,Xiao-qiang ZHAO,Yong-yong HUI,Hong-mei JIANG. Image super-resolution reconstruction algorithm based on multi-level continuous encoding and decoding. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1885-1893.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.09.020    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I9/1885

图 1  多级连续编码与解码的图像超分辨率重建算法的模型框架
图 2  2种编码与解码模型的对比示意图
图 3  多级连续编码与解码模型示意图
图 4  多级连续编码与解码的注意力残差模块结构示意图
算法 Set5 Set14 BSD100 Urban100
PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM
Bicubic 33.66 0.929 9 30.24 0.868 8 29.56 0.843 1 26.88 0.840 3
SRCNN[13] 36.66 0.954 2 32.45 0.906 7 31.36 0.887 9 29.50 0.894 6
FSRCNN[14] 37.05 0.956 0 32.66 0.909 0 31.53 0.892 0 29.88 0.902 0
VDSR[16] 37.53 0.959 0 33.05 0.913 0 31.90 0.896 0 30.77 0.914 0
DBPN[30] 38.09 0.960 0 33.85 0.919 0 32.27 0.900 0 32.55 0.932 4
RFDN[31] 38.05 0.960 6 33.68 0.918 4 32.16 0.899 4 32.12 0.927 8
A2N[22] 38.06 0.960 8 33.75 0.919 4 32.22 0.900 2 32.43 0.931 1
本研究 38.17 0.961 0 33.94 0.920 8 32.28 0.901 0 32.74 0.934 2
表 1  不同图像重建算法在4个测试集上的性能评价指标（放大倍数为2）
算法 Set5 Set14 BSD100 Urban100
PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM
Bicubic 30.39 0.868 2 27.55 0.774 2 27.21 0.738 5 24.46 0.734 9
SRCNN[13] 32.75 0.909 0 29.30 0.821 5 28.41 0.786 3 26.24 0.798 9
FSRCNN[14] 33.18 0.914 0 29.37 0.824 0 28.53 0.791 0 26.43 0.808 0
VDSR[16] 33.67 0.921 0 29.78 0.832 0 28.83 0.799 0 27.14 0.829 0
RFDN[31] 34.41 0.927 3 30.34 0.842 0 29.09 0.805 0 28.21 0.852 5
A2N[22] 34.47 0.927 9 30.44 0.843 7 29.14 0.805 9 28.41 0.857 0
本研究 34.58 0.928 8 30.51 0.845 6 29.20 0.807 9 28.66 0.863 4
表 2  不同图像重建算法在4个测试集上的性能评价指标（放大倍数为3）
算法 Set5 Set14 BSD100 Urban100
PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM PSNR/dB SSIM
Bicubic 28.42 0.810 4 26.00 0.702 7 25.96 0.667 5 23.14 0.657 7
SRCNN[13] 30.48 0.862 8 27.50 0.751 3 26.90 0.710 1 24.52 0.722 1
FSRCNN[14] 30.72 0.866 0 27.61 0.755 0 26.98 0.715 0 24.62 0.728 0
VDSR[16] 31.35 0.883 0 28.02 0.768 0 27.29 0.725 1 25.18 0.754 0
DBPN[30] 32.47 0.898 0 28.82 0.786 0 27.72 0.740 0 26.38 0.794 6
RFDN[31] 32.24 0.895 2 28.61 0.781 9 27.57 0.736 0 26.11 0.785 8
A2N[22] 32.30 0.896 6 28.71 0.784 2 27.61 0.737 4 26.27 0.792 0
本研究 32.37 0.897 1 28.76 0.785 7 27.55 0.734 0 26.41 0.797 4
表 3  不同图像重建算法在4个测试集上的性能评价指标（放大倍数为4）
图 5  不同图像重建算法重建的img004对比图像（放大倍数为4）
图 6  不同图像重建算法重建的img061对比图像（放大倍数为4）
图 7  不同图像重建算法重建的img096对比图像（放大倍数为4）
网络 PSNR/dB
未组合ED1、ED2、ED3、Attention 31.57
ED1 31.64
ED1+ED2 32.01
ED1+ED2+ED3 32.16
ED1+ED2+ED3+Attention 32.37
表 4  多级连续编码与解码的注意力残差模块消融研究结果(放大倍数为4)
图 8  不同图像重建算法在Urban100测试集上的性能对比(放大倍数为4)
20 ZHANG Y, TIAN Y, KONG Y, et al. Residual dense network for image super-resolution [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2472-2481.
21 ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks [C]// European Conference on Computer Vision. [S.l.]: Springer, 2018: 294-301.
22 CHEN H, GU J, ZHANG Z. Attention in attention network for image super-resolution [EB/OL]. [2022-04-21]. https://arxiv.org/pdf/2104.09497.pdf.
23 AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 126-135.
24 BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding [C]// Proceedings British Machine Vision Conference. Surrey: [s.n.], 2012: 1-10.
25 ZEYDE R, ELAD M, PROTTER M, et al. On single image scale-up using sparse-representations [C]// International Conference on Curves and Surfaces. [S.l.]: Springer, 2010: 711-730.
26 MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics [C]// Proceedings of the IEEE International Conference on Computer Vision. Vancouver: IEEE, 2001: 416-423.
27 HUANG J, SINGH A, AHUJA N, et al. Single image super-resolution from transformed self-exemplars [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 5197-5206
28 KINGMA D P, BA J L. ADAM: a method for stochastic optimization [EB/OL]. [2022-04-21]. https://arxiv.org/pdf/1412.6980.pdf.
29 WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612
doi: 10.1109/TIP.2003.819861
30 HARIS M, SHAKHNAROVICH G, UKITA N Deep back-projection networks for super-resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (12): 4323- 4337
doi: 10.1109/TPAMI.2020.3002836
31 LIU J, TANG J, WU G. Residual feature distillation network for lightweight image super-resolution [C]// European Conference on Computer Vision. [S.l.]: Springer, 2020: 41-55.
1 CHEN H, HE X, QING L, et al Real-world single image super-resolution: a brief review[J]. Information Fusion, 2022, 79: 124- 145
doi: 10.1016/j.inffus.2021.09.005
2 REBECQ H, RANFTL R, KOLTUN V, et al. Events-to-video: bringing modern computer vision to event cameras [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 3857-3866.
3 WANG Z, JIANG K, YI P, et al Ultra-dense GAN for satellite imagery super-resolution[J]. Neurocomputing, 2020, 398: 328- 373
doi: 10.1016/j.neucom.2019.03.106
4 LIU H, GU Y, WANG T, et al Satellite video super-resolution based on adaptively spatiotemporal neighbors and nonlocal similarity regularization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58 (12): 8372- 8383
doi: 10.1109/TGRS.2020.2987400
5 YUE L, SHEN H, LI J, et al Image super-resolution: the techniques, applications, and future[J]. Signal Processing, 2016, 128: 389- 408
doi: 10.1016/j.sigpro.2016.05.002
6 DENG L, YU D. Deep learning: methods and applications [M]. [S.l.]: Now Foundations and Trends, 2014: 197-387.
7 JORDAN M I, MITCHELL T M Machine learning: trends, perspectives, and prospects[J]. Science, 2015, 349 (6245): 255- 260
doi: 10.1126/science.aaa8415
8 LECUN Y, BENGIO Y, HINTON G Deep learning[J]. Nature, 2015, 521: 436- 444
doi: 10.1038/nature14539
9 ANWAR S, KHAN S, BARNES N A deep journey into super-resolution: a survey[J]. ACM Computing Surveys, 2020, 53 (3): 1- 34
10 WANG Z, CHEN J, HOI S C H Deep learning for image super-resolution: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (10): 3365- 3387
doi: 10.1109/TPAMI.2020.2982166
11 SONG Z, ZHAO X, JIANG H Gradual deep residual network for super-resolution[J]. Multimedia Tools and Applications, 2021, 80: 9765- 9778
doi: 10.1007/s11042-020-10152-9
12 SONG Z, ZHAO X, HUI Y, et al Progressive back-projection network for COVID-CT super-resolution[J]. Computer Methods and Programs in Biomedicine, 2021, 208: 106193
doi: 10.1016/j.cmpb.2021.106193
13 DONG C, LOY C C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38 (2): 295- 307
14 DONG C, LOY C C, TANG X, et al. Accelerating the super-resolution convolutional neural network [C]// European Conference on Computer Vision. [S.l.]: Springer, 2016: 391-407.
15 SHI W, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
16 KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646-1654.
17 KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1637-1645.
18 LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 136-144.
[1] 李海烽,张雪英,段淑斐,贾海蓉,Huizhi Liang . 融合生成对抗网络与时间卷积网络的普通话情感识别[J]. 浙江大学学报(工学版), 2023, 57(9): 1865-1875.
[2] 赵小强,王泽,宋昭漾,蒋红梅. 基于动态注意力网络的图像超分辨率重建[J]. 浙江大学学报(工学版), 2023, 57(8): 1487-1494.
[3] 王慧欣,童向荣. 融合知识图谱的推荐系统研究进展[J]. 浙江大学学报(工学版), 2023, 57(8): 1527-1540.
[4] 王殿海,谢瑞,蔡正义. 基于最优汇集时间间隔的城市间断交通流预测[J]. 浙江大学学报(工学版), 2023, 57(8): 1607-1617.
[5] 宋秀兰,董兆航,单杭冠,陆炜杰. 基于时空融合的多头注意力车辆轨迹预测[J]. 浙江大学学报(工学版), 2023, 57(8): 1636-1643.
[6] 李晓艳,王鹏,郭嘉,李雪,孙梦宇. 基于双注意力机制的多分支孪生网络目标跟踪[J]. 浙江大学学报(工学版), 2023, 57(7): 1307-1316.
[7] 李云红,段姣姣,苏雪平,张蕾涛,于惠康,刘杏瑞. 基于改进生成对抗网络的书法字生成算法[J]. 浙江大学学报(工学版), 2023, 57(7): 1326-1334.
[8] 刘春娟,乔泽,闫浩文,吴小所,王嘉伟,辛钰强. 基于多尺度互注意力的遥感图像语义分割网络[J]. 浙江大学学报(工学版), 2023, 57(7): 1335-1344.
[9] 权巍,蔡永青,王超,宋佳,孙鸿凯,李林轩. 基于3D-ResNet双流网络的VR病评估模型[J]. 浙江大学学报(工学版), 2023, 57(7): 1345-1353.
[10] 张艳,孙晶雪,孙叶美,刘树东,王传启. 基于分割注意力与线性变换的轻量化目标检测[J]. 浙江大学学报(工学版), 2023, 57(6): 1195-1204.
[11] 张海波,蔡磊,任俊平,王汝言,刘富. 基于Transformer的高效自适应语义分割网络[J]. 浙江大学学报(工学版), 2023, 57(6): 1205-1214.
[12] 韩俊,袁小平,王准,陈烨. 基于YOLOv5s的无人机密集小目标检测算法[J]. 浙江大学学报(工学版), 2023, 57(6): 1224-1233.
[13] 周传华,操礼春,周家亿,詹凤. 图卷积融合计算时效网络节点重要性评估分析[J]. 浙江大学学报(工学版), 2023, 57(5): 930-938.
[14] 王誉翔,钟智伟,夏鹏程,黄亦翔,刘成良. 基于改进Transformer的复合故障解耦诊断方法[J]. 浙江大学学报(工学版), 2023, 57(5): 855-864.
[15] 吕鑫栋,李娇,邓真楠,冯浩,崔欣桐,邓红霞. 基于改进Transformer的结构化图像超分辨网络[J]. 浙江大学学报(工学版), 2023, 57(5): 865-874.