Please wait a minute...
浙江大学学报(工学版)  2019, Vol. 53 Issue (7): 1331-1339    DOI: 10.3785/j.issn.1008-973X.2019.07.012
自动化技术、计算机技术     
基于多尺度特征映射网络的图像超分辨率重建
段然(),周登文*(),赵丽娟,柴晓亮
华北电力大学 控制与计算机工程学院,北京 102206
Image super-resolution reconstruction based on multi-scale feature mapping network
Ran DUAN(),Deng-wen ZHOU*(),Li-juan ZHAO,Xiao-liang CHAI
School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, China
 全文: PDF(1184 KB)   HTML
摘要:

针对基于卷积神经网络的图像超分辨率重建(SRCNN)方法存在的重建网络浅、特征利用率低以及重建图像模糊等问题,提出基于多尺度特征映射网络的图像超分辨率重建方法. 多尺度特征映射网络通过学习低分辨率(LR)特征与高分辨率(HR)特征之间的映射关系,将多个尺度的LR特征映射到HR特征空间,通过特征融合来提高重建过程中对特征的利用率;该方法定义了结合逐像素损失、感知损失和对抗损失的联合损失函数,从低频内容、图像边缘和局部纹理等方面均衡提升重建图像质量. 对数据集Set5、Set14和BSD100的图片4倍下采样后进行测试,与当前主流方法进行比较和分析. 实验证明,基于生成对抗的多尺度特征映射网络在提高图像感知质量方面表现优秀,重建的图像具有更加清晰的边缘和纹理,在客观评价上具有较好的评分.

关键词: 卷积神经网络超分辨率重建生成对抗网络深度学习感知损失    
Abstract:

An image super-resolution reconstruction method based on multi-scale feature mapping network was proposed for the problems of shallow network, low utilization rate of features and fuzzy reconstructed images, which existed in the super-resolution convolutional neural network (SRCNN). Multi-scale low-resolution (LR) features were mapped into high-resolution (HR) feature space by learning the mapping relation between LR features and HR features, and the utilization rate of features in the reconstruction process was improved by using feature concatenation. A joint loss function consisting of the pixel-wise loss, the perceptual loss and the adversarial loss was defined, which performed well in restoring the low-frequency content, the sharp edges and the high-frequency textures of the reconstructed images. The experimental results of datasets Set5, Set14 and BSD100 for the upscaling factor 4 were compared with those of state-of-the-art methods. The proposed method performs well in improving the perceptual quality of the reconstructed images in order to achieve clearer edges and textures, and has better scores in the objective evaluation.

Key words: convolutional neural network    super-resolution reconstruction    generative adversarial network    deep learning    perceptual loss
收稿日期: 2018-08-22 出版日期: 2019-06-25
CLC:  TP 391  
通讯作者: 周登文     E-mail: 1162227075@ncepu.edu.cn;zdw@ncepu.edu.cn
作者简介: 段然(1993—),女,硕士生,从事图像处理的研究. orcid.org/0000-0002-2778-1148. E-mail: dr1115262020@163.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
段然
周登文
赵丽娟
柴晓亮

引用本文:

段然,周登文,赵丽娟,柴晓亮. 基于多尺度特征映射网络的图像超分辨率重建[J]. 浙江大学学报(工学版), 2019, 53(7): 1331-1339.

Ran DUAN,Deng-wen ZHOU,Li-juan ZHAO,Xiao-liang CHAI. Image super-resolution reconstruction based on multi-scale feature mapping network. Journal of ZheJiang University (Engineering Science), 2019, 53(7): 1331-1339.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2019.07.012        http://www.zjujournals.com/eng/CN/Y2019/V53/I7/1331

图 1  生成器网络结构图
图 2  判别器网络结构图
图 3  使用不同映射模块的网络的重建结果对比
SR方法 PSNR/dB, SSIM
Set5 Set14 BSD100 平均值
双三次插值 28.43, 0.810 25.77, 0.703 25.98, 0.671 26.73, 0.728
A+[26] 30.30, 0.859 27.38, 0.752 26.82, 0.710 28.17, 0.774
SRCNN[6] 30.49, 0.863 27.50, 0.751 26.91, 0.712 28.30, 0.775
FSRCNN[8] 30.71, 0.865 27.60, 0.753 26.97, 0.713 28.43, 0.777
文献[10]方法 30.80, 0.873 28.60, 0.792 26.96, 0.711 28.78, 0.792
本文方法(MSE) 31.32, 0.881 27.78, 0.773 27.40, 0.736 28.83, 0.797
表 1  4倍放大因子下与其他5种SR方法在Set5、Set14和BSD100上的平均PSNR和SSIM对比
图 4  在4倍放大因子下与双三次插值、A+、SRCNN、FSRCNN和文献[10]方法的重建结果对比图
图 5  多尺度特征映射网络基于不同损失函数的重建图像对比
图 6  在4倍放大因子上对Set14中的Comic图片的超分辨率重建结果对比图
图 7  在4倍放大因子上对Set14中的Lenna图片的超分辨率重建结果对比图
图 8  在4倍放大因子上对Set5中的Baby图片的超分辨率重建结果对比图
SR方法 PSNR/dB, SSIM
Set5 Set14 BSD100 平均值
文献[14]方法 27.09, 0.768 24.99, 0.673 24.95, 0.631 25.68, 0.691
SRGAN[15] 29.36, 0.833 25.96, 0.696 25.20, 0.642 26.84, 0.724
文献[17]方法 31.30, 0.877 27.72, 0.755 26.65, 0.703 28.56, 0.778
本文方法(EPA) 29.67, 0.887 26.63, 0.801 26.16, 0.774 27.49, 0.821
表 2  在4倍放大因子下与文献[14]方法、SRGAN以及文献[17]方法的平均PSNR和平均SSIM对比
1 YUE L W, SHEN H F, LI J, et al Image super-resolution: the techniques, applications, and future[J]. Signal Processing, 2016, 128 (11): 389- 408
2 陆志芳, 钟宝江 基于预测梯度的图像插值算法[J]. 自动化学报, 2018, 44 (06): 1072- 1085
LU Zhi-fang, ZHONG Bao-jiang Image interpolation with predicted gradients[J]. Acta Automatica Sinica, 2018, 44 (06): 1072- 1085
3 孙京, 袁强强, 李冀玮, 等 亮度--梯度联合约束的车牌图像超分辨率重建[J]. 中国图象图形学报, 2018, 23 (06): 802- 813
SUN Jing, YUAN Qiang-qiang, LI Ji-wei, et al License plate image super-resolution based on intensity-gradient prior combination[J]. Journal of Image and Graphics, 2018, 23 (06): 802- 813
4 YANG J, WRIGHT J, HUANG T, et al. Image super-resolution as sparse representation of raw image patches [C] // IEEE Conference on Computer Vision and Pattern Recognition. Anchorage: IEEE, 2008: 1–8.
5 孙旭, 李晓光, 李嘉锋, 等 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43 (5): 697- 709
SUN Xu, LI Xiao-guang, LI Jia-feng, et al Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43 (5): 697- 709
6 DONG C, CHEN C L, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (2): 295- 307
doi: 10.1109/TPAMI.2015.2439281
7 YANG S, SUN Y, CHEN Y, et al Structural similarity regularized and sparse coding based super-resolution for medical images[J]. Biomedical Signal Processing and Control, 2012, 7 (6): 579- 590
doi: 10.1016/j.bspc.2012.08.001
8 DONG C, CHEN C L, TANG X. Accelerating the super-resolution convolutional neural network [C] // European Conference on Computer Vision. Amsterdam: Springer, 2016: 391–407.
9 KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks [C] // IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646–1654.
10 谢珍珠, 吴从中, 詹曙 边缘增强深层网络的图像超分辨率重建[J]. 中国图象图形学报, 2018, 23 (1): 114- 122
XIE Zhen-zhu, WU Cong-zhong, ZHAN Shu Image super-resolution reconstruction via deep network based on edge-enhancement[J]. Journal of Image and Graphics, 2018, 23 (1): 114- 122
11 李云飞, 符冉迪, 金炜, 等 多通道卷积的图像超分辨率方法[J]. 中国图象图形学报, 2017, 22 (12): 1690- 1700
LI Yun-fei, FU Ran-di, JIN Wei, et al Image super-resolution using multi-channel convolution[J]. Journal of Image and Graphics, 2017, 22 (12): 1690- 1700
12 陈书贞, 解小会, 杨郁池, 等 利用多尺度卷积神经网络的图像超分辨率算法[J]. 信号处理, 2018, 34 (09): 1033- 1044
CHEN Shu-zhen, XIE Xiao-hui, YANG Yu-chi, et al Image super-resolution algorithm based on multi-scale convolution neural network[J]. Journal of Signal Processing, 2018, 34 (09): 1033- 1044
13 GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks [C] // IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2414–2423.
14 JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution [C] // European Conference on Computer Vision. Amsterdam: Springer, 2016: 694–711.
15 LEDIG C, WANG Z, SHI W, et al. Photo-realistic single image super-resolution using a generative adversarial network [C] // IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 105–114.
16 GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets [C] // International Conference on Neural Information Processing Systems. Montreal: ACM, 2014: 2672–2680.
17 DENG X Enhancing image quality via style transfer for single image super-resolution[J]. IEEE Signal Processing Letters, 2018, 25 (4): 571- 575
doi: 10.1109/LSP.2018.2805809
18 SHI W, CABALLERO J, HUSZáR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C] // IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874–1883.
19 REED S, AKATA Z, YAN X, et al. Generative adversarial text to image synthesis [C] // International Conference on International Conference on Machine Learning. New York City: ACM, 2016: 1060–1069.
20 ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C] // IEEE Conference on Computer Vision and Pattern Recognition. Hawaii, USA: IEEE, 2017: 5967–5976.
21 SAJJADI M S, SCHOLKOPF B, HIRSCH M, et al. EnhanceNet: single image super-resolution through automated texture synthesis [C] // IEEE International Conference on Computer Vision. Zurich: IEEE, 2017: 4501–4510.
22 LI C, WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks [C] // European Conference on Computer Vision. Amsterdam: Springer, 2016: 702–716.
23 YANG J, WRIGHT J, HUANG T S, et al Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19 (11): 2861- 2873
doi: 10.1109/TIP.2010.2050625
24 TIMOFTE R, LEE K M, WANG X, et al. NTIRE 2017 challenge on single image super-resolution: methods and results [C] // IEEE Conference on Computer Vision and Pattern Recognition Workshops. Hawaii, USA: IEEE, 2017: 1110–1121.
25 HE K, ZHANG X, REN S, et al. Delving deep into rectifiers: surpassing human-level performance on imageNet classification [C] // IEEE International Conference on Computer Vision. Santiago: IEEE, 2015:1026–1034.
[1] 许佳辉,王敬昌,陈岭,吴勇. 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报(工学版), 2021, 55(4): 601-607.
[2] 王虹力,郭斌,刘思聪,刘佳琪,仵允港,於志文. 边端融合的终端情境自适应深度感知模型[J]. 浙江大学学报(工学版), 2021, 55(4): 626-638.
[3] 张腾,蒋鑫龙,陈益强,陈前,米涛免,陈彪. 基于腕部姿态的帕金森病用药后开-关期检测[J]. 浙江大学学报(工学版), 2021, 55(4): 639-647.
[4] 郑英杰,吴松荣,韦若禹,涂振威,廖进,刘东. 基于目标图像FCM算法的地铁定位点匹配及误报排除方法[J]. 浙江大学学报(工学版), 2021, 55(3): 586-593.
[5] 徐利锋,黄海帆,丁维龙,范玉雷. 基于改进DenseNet的水果小目标检测[J]. 浙江大学学报(工学版), 2021, 55(2): 377-385.
[6] 胡惠雅,盖绍彦,达飞鹏. 基于生成对抗网络的偏转人脸转正[J]. 浙江大学学报(工学版), 2021, 55(1): 116-123.
[7] 许豪灿,李基拓,陆国栋. 由LeNet-5从单张着装图像重建三维人体[J]. 浙江大学学报(工学版), 2021, 55(1): 153-161.
[8] 黄毅鹏,胡冀苏,钱旭升,周志勇,赵文露,马麒,沈钧康,戴亚康. SE-Mask-RCNN:多参数MRI前列腺癌分割方法[J]. 浙江大学学报(工学版), 2021, 55(1): 203-212.
[9] 陈巧红,陈翊,李文书,贾宇波. 多尺度SE-Xception服装图像分类[J]. 浙江大学学报(工学版), 2020, 54(9): 1727-1735.
[10] 郑浦,白宏阳,李伟,郭宏伟. 复杂背景下的小目标检测算法[J]. 浙江大学学报(工学版), 2020, 54(9): 1777-1784.
[11] 周登文,田金月,马路遥,孙秀秀. 基于多级特征并联的轻量级图像语义分割[J]. 浙江大学学报(工学版), 2020, 54(8): 1516-1524.
[12] 张彦楠,黄小红,马严,丛群. 基于深度学习的录音文本分类方法[J]. 浙江大学学报(工学版), 2020, 54(7): 1264-1271.
[13] 明涛,王丹,郭继昌,李锵. 基于多尺度通道重校准的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2020, 54(7): 1289-1297.
[14] 闫旭,范晓亮,郑传潘,臧彧,王程,程明,陈龙彪. 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报(工学版), 2020, 54(6): 1147-1155.
[15] 沈宗礼,余建波. 基于迁移学习与深度森林的晶圆图缺陷识别[J]. 浙江大学学报(工学版), 2020, 54(6): 1228-1239.