Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2019, Vol. 53 Issue (7): 1331-1339    DOI: 10.3785/j.issn.1008-973X.2019.07.012
Automatic Technology, Computer Technology     
Image super-resolution reconstruction based on multi-scale feature mapping network
Ran DUAN(),Deng-wen ZHOU*(),Li-juan ZHAO,Xiao-liang CHAI
School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, China
Download: HTML     PDF(1184KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

An image super-resolution reconstruction method based on multi-scale feature mapping network was proposed for the problems of shallow network, low utilization rate of features and fuzzy reconstructed images, which existed in the super-resolution convolutional neural network (SRCNN). Multi-scale low-resolution (LR) features were mapped into high-resolution (HR) feature space by learning the mapping relation between LR features and HR features, and the utilization rate of features in the reconstruction process was improved by using feature concatenation. A joint loss function consisting of the pixel-wise loss, the perceptual loss and the adversarial loss was defined, which performed well in restoring the low-frequency content, the sharp edges and the high-frequency textures of the reconstructed images. The experimental results of datasets Set5, Set14 and BSD100 for the upscaling factor 4 were compared with those of state-of-the-art methods. The proposed method performs well in improving the perceptual quality of the reconstructed images in order to achieve clearer edges and textures, and has better scores in the objective evaluation.



Key wordsconvolutional neural network      super-resolution reconstruction      generative adversarial network      deep learning      perceptual loss     
Received: 22 August 2018      Published: 25 June 2019
CLC:  TP 391  
Corresponding Authors: Deng-wen ZHOU     E-mail: 1162227075@ncepu.edu.cn;zdw@ncepu.edu.cn
Cite this article:

Ran DUAN,Deng-wen ZHOU,Li-juan ZHAO,Xiao-liang CHAI. Image super-resolution reconstruction based on multi-scale feature mapping network. Journal of ZheJiang University (Engineering Science), 2019, 53(7): 1331-1339.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2019.07.012     OR     http://www.zjujournals.com/eng/Y2019/V53/I7/1331


基于多尺度特征映射网络的图像超分辨率重建

针对基于卷积神经网络的图像超分辨率重建(SRCNN)方法存在的重建网络浅、特征利用率低以及重建图像模糊等问题,提出基于多尺度特征映射网络的图像超分辨率重建方法. 多尺度特征映射网络通过学习低分辨率(LR)特征与高分辨率(HR)特征之间的映射关系,将多个尺度的LR特征映射到HR特征空间,通过特征融合来提高重建过程中对特征的利用率;该方法定义了结合逐像素损失、感知损失和对抗损失的联合损失函数,从低频内容、图像边缘和局部纹理等方面均衡提升重建图像质量. 对数据集Set5、Set14和BSD100的图片4倍下采样后进行测试,与当前主流方法进行比较和分析. 实验证明,基于生成对抗的多尺度特征映射网络在提高图像感知质量方面表现优秀,重建的图像具有更加清晰的边缘和纹理,在客观评价上具有较好的评分.


关键词: 卷积神经网络,  超分辨率重建,  生成对抗网络,  深度学习,  感知损失 
Fig.1 Structure of generator network
Fig.2 Structure of discriminator network
Fig.3 Comparison with reconstruction results of networks using different mapping modules (4×)
SR方法 PSNR/dB, SSIM
Set5 Set14 BSD100 平均值
双三次插值 28.43, 0.810 25.77, 0.703 25.98, 0.671 26.73, 0.728
A+[26] 30.30, 0.859 27.38, 0.752 26.82, 0.710 28.17, 0.774
SRCNN[6] 30.49, 0.863 27.50, 0.751 26.91, 0.712 28.30, 0.775
FSRCNN[8] 30.71, 0.865 27.60, 0.753 26.97, 0.713 28.43, 0.777
文献[10]方法 30.80, 0.873 28.60, 0.792 26.96, 0.711 28.78, 0.792
本文方法(MSE) 31.32, 0.881 27.78, 0.773 27.40, 0.736 28.83, 0.797
Tab.1 Average PSNRs and SSIMs compared with other SR methods on Set5,Set14 and BSD100 (4×)
Fig.4 Reconstruction results compared with bicubic interpolation,A+,SRCNN,FSRCNN and reference [10] (4×)
Fig.5 Comparison of multi-scale feature mapping networks based on different loss functions
Fig.6 Comparison of reconstructed Comic image in Set14 on ×4 upscaling
Fig.7 Comparison of reconstructed Lenna image in Set14 on ×4 upscaling
Fig.8 Comparison of reconstructed Baby image in Set5 on ×4 upscaling
SR方法 PSNR/dB, SSIM
Set5 Set14 BSD100 平均值
文献[14]方法 27.09, 0.768 24.99, 0.673 24.95, 0.631 25.68, 0.691
SRGAN[15] 29.36, 0.833 25.96, 0.696 25.20, 0.642 26.84, 0.724
文献[17]方法 31.30, 0.877 27.72, 0.755 26.65, 0.703 28.56, 0.778
本文方法(EPA) 29.67, 0.887 26.63, 0.801 26.16, 0.774 27.49, 0.821
Tab.2 Average PSNRs and SSIMs compared with reference [14],SRGAN and reference [17] method (4×)
[1]   YUE L W, SHEN H F, LI J, et al Image super-resolution: the techniques, applications, and future[J]. Signal Processing, 2016, 128 (11): 389- 408
[2]   陆志芳, 钟宝江 基于预测梯度的图像插值算法[J]. 自动化学报, 2018, 44 (06): 1072- 1085
LU Zhi-fang, ZHONG Bao-jiang Image interpolation with predicted gradients[J]. Acta Automatica Sinica, 2018, 44 (06): 1072- 1085
[3]   孙京, 袁强强, 李冀玮, 等 亮度--梯度联合约束的车牌图像超分辨率重建[J]. 中国图象图形学报, 2018, 23 (06): 802- 813
SUN Jing, YUAN Qiang-qiang, LI Ji-wei, et al License plate image super-resolution based on intensity-gradient prior combination[J]. Journal of Image and Graphics, 2018, 23 (06): 802- 813
[4]   YANG J, WRIGHT J, HUANG T, et al. Image super-resolution as sparse representation of raw image patches [C] // IEEE Conference on Computer Vision and Pattern Recognition. Anchorage: IEEE, 2008: 1–8.
[5]   孙旭, 李晓光, 李嘉锋, 等 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43 (5): 697- 709
SUN Xu, LI Xiao-guang, LI Jia-feng, et al Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43 (5): 697- 709
[6]   DONG C, CHEN C L, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (2): 295- 307
doi: 10.1109/TPAMI.2015.2439281
[7]   YANG S, SUN Y, CHEN Y, et al Structural similarity regularized and sparse coding based super-resolution for medical images[J]. Biomedical Signal Processing and Control, 2012, 7 (6): 579- 590
doi: 10.1016/j.bspc.2012.08.001
[8]   DONG C, CHEN C L, TANG X. Accelerating the super-resolution convolutional neural network [C] // European Conference on Computer Vision. Amsterdam: Springer, 2016: 391–407.
[9]   KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks [C] // IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646–1654.
[10]   谢珍珠, 吴从中, 詹曙 边缘增强深层网络的图像超分辨率重建[J]. 中国图象图形学报, 2018, 23 (1): 114- 122
XIE Zhen-zhu, WU Cong-zhong, ZHAN Shu Image super-resolution reconstruction via deep network based on edge-enhancement[J]. Journal of Image and Graphics, 2018, 23 (1): 114- 122
[11]   李云飞, 符冉迪, 金炜, 等 多通道卷积的图像超分辨率方法[J]. 中国图象图形学报, 2017, 22 (12): 1690- 1700
LI Yun-fei, FU Ran-di, JIN Wei, et al Image super-resolution using multi-channel convolution[J]. Journal of Image and Graphics, 2017, 22 (12): 1690- 1700
[12]   陈书贞, 解小会, 杨郁池, 等 利用多尺度卷积神经网络的图像超分辨率算法[J]. 信号处理, 2018, 34 (09): 1033- 1044
CHEN Shu-zhen, XIE Xiao-hui, YANG Yu-chi, et al Image super-resolution algorithm based on multi-scale convolution neural network[J]. Journal of Signal Processing, 2018, 34 (09): 1033- 1044
[13]   GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks [C] // IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2414–2423.
[14]   JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution [C] // European Conference on Computer Vision. Amsterdam: Springer, 2016: 694–711.
[15]   LEDIG C, WANG Z, SHI W, et al. Photo-realistic single image super-resolution using a generative adversarial network [C] // IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 105–114.
[16]   GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets [C] // International Conference on Neural Information Processing Systems. Montreal: ACM, 2014: 2672–2680.
[17]   DENG X Enhancing image quality via style transfer for single image super-resolution[J]. IEEE Signal Processing Letters, 2018, 25 (4): 571- 575
doi: 10.1109/LSP.2018.2805809
[18]   SHI W, CABALLERO J, HUSZáR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C] // IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874–1883.
[19]   REED S, AKATA Z, YAN X, et al. Generative adversarial text to image synthesis [C] // International Conference on International Conference on Machine Learning. New York City: ACM, 2016: 1060–1069.
[20]   ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C] // IEEE Conference on Computer Vision and Pattern Recognition. Hawaii, USA: IEEE, 2017: 5967–5976.
[21]   SAJJADI M S, SCHOLKOPF B, HIRSCH M, et al. EnhanceNet: single image super-resolution through automated texture synthesis [C] // IEEE International Conference on Computer Vision. Zurich: IEEE, 2017: 4501–4510.
[22]   LI C, WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks [C] // European Conference on Computer Vision. Amsterdam: Springer, 2016: 702–716.
[23]   YANG J, WRIGHT J, HUANG T S, et al Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19 (11): 2861- 2873
doi: 10.1109/TIP.2010.2050625
[24]   TIMOFTE R, LEE K M, WANG X, et al. NTIRE 2017 challenge on single image super-resolution: methods and results [C] // IEEE Conference on Computer Vision and Pattern Recognition Workshops. Hawaii, USA: IEEE, 2017: 1110–1121.
[25]   HE K, ZHANG X, REN S, et al. Delving deep into rectifiers: surpassing human-level performance on imageNet classification [C] // IEEE International Conference on Computer Vision. Santiago: IEEE, 2015:1026–1034.
[1] Jia-hui XU,Jing-chang WANG,Ling CHEN,Yong WU. Surface water quality prediction model based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 601-607.
[2] Hong-li WANG,Bin GUO,Si-cong LIU,Jia-qi LIU,Yun-gang WU,Zhi-wen YU. End context-adaptative deep sensing model with edge-end collaboration[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 626-638.
[3] Teng ZHANG,Xin-long JIANG,Yi-qiang CHEN,Qian CHEN,Tao-mian MI,Piu CHAN. Wrist attitude-based Parkinson's disease ON/OFF state assessment after medication[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 639-647.
[4] Ying-jie ZHENG,Song-rong WU,Ruo-yu WEI,Zhen-wei TU,Jin LIAO,Dong LIU. Metro location point matching and false alarm elimination based on FCM algorithm of target image[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 586-593.
[5] Li-feng XU,Hai-fan HUANG,Wei-long DING,Yu-lei FAN. Detection of small fruit target based on improved DenseNet[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(2): 377-385.
[6] Hui-ya HU,Shao-yan GAI,Fei-peng DA. Face frontalization based on generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 116-123.
[7] Hao-can XU,Ji-tuo LI,Guo-dong LU. Reconstruction of three-dimensional human bodies from single image by LeNet-5[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 153-161.
[8] Yi-peng HUANG,Ji-su HU,Xu-sheng QIAN,Zhi-yong ZHOU,Wen-lu ZHAO,Qi MA,Jun-kang SHEN,Ya-kang DAI. SE-Mask-RCNN: segmentation method for prostate cancer on multi-parametric MRI[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 203-212.
[9] Qiao-hong CHEN,YI CHEN,Wen-shu Li,Yu-bo JIA. Clothing image classification based on multi-scale SE-Xception[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1727-1735.
[10] Pu ZHENG,Hong-yang BAI,Wei LI,Hong-wei GUO. Small target detection algorithm in complex background[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1777-1784.
[11] Deng-wen ZHOU,Jin-yue TIAN,Lu-yao MA,Xiu-xiu SUN. Lightweight image semantic segmentation based on multi-level feature cascaded network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(8): 1516-1524.
[12] Yan-nan ZHANG,Xiao-hong HUANG,Yan MA,Qun CONG. Method with recording text classification based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1264-1271.
[13] Tao MING,Dan WANG,Ji-chang GUO,Qiang LI. Breast cancer histopathological image classification using multi-scale channel squeeze-and-excitation model[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1289-1297.
[14] Xu YAN,Xiao-liang FAN,Chuan-pan ZHENG,Yu ZANG,Cheng WANG,Ming CHENG,Long-biao CHEN. Urban traffic flow prediction algorithm based on graph convolutional neural networks[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(6): 1147-1155.
[15] Hai-jin WANG,Zong-yu YIN,Zhen-zheng KE,Ying-jie GUO,Hui-yue DONG. Wear monitoring of helical milling tool based on one-dimensional convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(5): 931-939.