Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2021, Vol. 55 Issue (8): 1510-1517    DOI: 10.3785/j.issn.1008-973X.2021.08.012
计算机技术     
基于纹理修复的双焦相机连续数字变焦算法
宋炯辉1(),李奇1,*(),王静2,徐之海1,冯华君1,陈跃庭1
1. 浙江大学 光电科学与工程学院，浙江 杭州 310027
2. 光学辐射重点实验室，北京 100854
Continuous digital zooming algorithm of dual-focal camera based on texture restoration
Jiong-hui SONG1(),Qi LI1,*(),Jing WANG2,Zhi-hai XU1,Hua-jun FENG1,Yue-ting CHEN1
1. College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China
2. Science and Technology on Optical Radiation Laboratory, Beijing 100854, China
 全文: PDF(1237 KB)   HTML
摘要：

设计主要用于双焦相机成像系统的连续数字变焦算法. 该算法将双焦相机连续数字变焦问题拆分成长焦相机视场内的特征迁移问题和长焦相机视场外的纹理修复问题. 在实现细节上，该算法参考基于上下文语义的图像修复算法的思路，利用长焦相机图像的纹理信息修复短焦相机图像，并在2个问题上使用相似的网络结构来降低长焦相机视场内外细节视觉效果的差异. 实验结果表明，相比其他算法，所提算法能显著提升变焦图像的质量，有效改善长焦相机视场内外纹理细节差距较大的问题，在主客观评价上都具有更好的表现. 此外，该方法对于处于长焦相机视场外的纹理细节，具有明显的修复效果，对于实际拍摄的图像同样有效.
关键词： 双焦相机;  连续数字变焦;  上下文语义;  纹理修复;  图像修复    
Abstract:

A continuous digital zooming algorithm mainly for dual-focal camera imaging system was proposed. The continuous digital zooming problem of dual-focal camera was divided into the feature transfer problem in the field of view of the long-focus camera and the texture repair problem outside the field of view of the long-focus camera. The texture information of long-focus camera image was used to restore short-focus camera image referring to contextual image restoration algorithm, and a similar network structure was used to reduce the visual difference inside and outside the field of view of the long-focus camera. Experimental results show that compared with other algorithms, the proposed algorithm can improve the quality of images, improve the problem of large gap between the texture details inside and outside the field of view of the long-focus camera, and has higher subjective resolution and better visual clarity. In addition, the algorithm has an obvious repair effect on the texture details outside the field of view of the long-focus camera and also performs well for the actual images.
Key words: dual-focal camera    continuous digital zooming    context    texture restoration    image restoration
收稿日期: 2020-08-03 出版日期: 2021-09-01
CLC:  TP 391  
基金资助: 装备预研重点实验室基金资助项目（61424080214）；十三五民用航天资助项目
通讯作者: 李奇     E-mail: 21830051@zju.edu.cn;liqi@zju.edu.cn
作者简介: 宋炯辉（1996—），男，硕士生，从事数字图像处理研究. orcid.org/0000-0002-1848-0116. E-mail: 21830051@zju.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
宋炯辉
李奇
王静
徐之海
冯华君
陈跃庭

引用本文:

宋炯辉,李奇,王静,徐之海,冯华君,陈跃庭. 基于纹理修复的双焦相机连续数字变焦算法[J]. 浙江大学学报(工学版), 2021, 55(8): 1510-1517.

Jiong-hui SONG,Qi LI,Jing WANG,Zhi-hai XU,Hua-jun FENG,Yue-ting CHEN. Continuous digital zooming algorithm of dual-focal camera based on texture restoration. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1510-1517.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2021.08.012    或    https://www.zjujournals.com/eng/CN/Y2021/V55/I8/1510

图 1  双焦相机成像系统示意图
图 2  本研究细节修复算法的流程图
图 3  纹理修复子网络示意图
图 4  数据集中的一组训练集图像
图 5  不同算法连续数字变焦的仿真实验结果
图 6  不同图像不同变焦倍率的仿真实验结果
算法 ×1.2 ×1.4 ×1.6 ×1.8 ×2.0 ×2.2 ×2.4 ×2.6 ×2.8 ×3.0 ×3.2 ×3.4 ×3.6 ×3.8
双三次插值 35.01 32.93 31.49 30.38 29.55 28.87 28.31 27.89 27.51 27.19 26.89 26.60 26.35 26.07
TTSR[11] 39.79 36.73 34.53 33.27 32.15 31.29 30.56 29.98 29.53 29.19 28.87 28.55 28.29 27.96
文献[9]算法 39.11 36.29 34.05 33.11 32.03 31.07 30.19 29.53 29.02 28.68 28.22 27.89 27.43 27.18
本研究算法 39.46 36.33 34.18 33.25 32.32 31.56 30.89 30.17 29.71 29.43 29.06 28.77 28.50 28.05
表 1  不同算法不同变焦倍率的峰值信噪比
视场 算法 ×1.5 ×2.0 ×2.5 ×3.0 ×3.5
长焦相机视场内 双三次插值 32.03 29.81 28.06 27.12 26.49
TTSR[11] 37.57 34.23 32.04 30.41 29.88
文献[9]算法 36.56 33.44 31.10 29.39 28.53
本研究算法 37.75 34.21 31.93 30.28 29.71
长焦相机视场外 双三次插值 32.25 29.42 27.94 27.26 26.47
TTSR[11] 35.21 31.45 29.08 27.62 27.93
文献[9]算法 34.80 31.54 29.01 27.76 27.31
本研究算法 34.72 31.69 29.58 28.33 28.24
表 2  不同算法长焦相机视场内外的峰值信噪比
图 7  长焦相机视场外纹理修复效果对比
图 8  过渡区域视觉效果对比
图 9  真实图像实验结果
1 KEYS R Cubic convolution interpolation for digital image processing[J]. Speech and Signal Processing, 1981, 29 (6): 1153- 1160
doi: 10.1109/TASSP.1981.1163711
2 LI X New edge-directed interpolation[J]. IEEE Transactions on Image Processing, 2001, 10 (10): 1521- 1527
doi: 10.1109/83.951537
3 DONG C, LOY C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (2): 295- 307
doi: 10.1109/TPAMI.2015.2439281
4 KIM J, LEE J, LEE K. Accurate image super-resolution using very deep convolutional networks[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 1646-1654.
5 LEDIG C, THEIS L, HUSZAR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hawaii: IEEE, 2017: 4681-4690.
6 YU S, MOON B, KIM D, et al Continuous digital zooming of asymmetric dual camera images using registration and variational image restoration[J]. Multidimensional Systems and Signal Processing, 2018, 29 (4): 1959- 1987
doi: 10.1007/s11045-017-0534-4
7 MOON B, YU S, KO S, et al Continuous digital zooming using local self-similarity-based superresolution for an asymmetric dual camera system[J]. Journal of the Optical Society of America A-optics Image Science and Vision, 2017, 34 (6): 991- 1003
doi: 10.1364/JOSAA.34.000991
8 MA H, LI Q, XU Z, et al Photo-realistic continuous digital zooming for an asymmetrical dual camera system[J]. Optics and Laser Technology, 2019, 109: 110- 122
doi: 10.1016/j.optlastec.2018.07.056
9 赫贵然, 李奇, 冯华君, 等 基于CNN特征提取的双焦相机连续数字变焦[J]. 浙江大学学报: 工学版, 2019, 53 (6): 1182- 1189
HE Gui-ran, LI Qi, FENG Hua-jun, et al Dual-focal camera continuous digital zoom based on CNN and feature extraction[J]. Journal of Zhejiang University: Engineering Science, 2019, 53 (6): 1182- 1189
10 ZHANG Z, WANG Z, LIN Z, et al. Image super-resolution by neural texture transfer[C]// 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Long Bench: IEEE, 2019: 7982-7991.
11 YANG F, YANG H, FU J, et al. Learning texture transformer network for image super-resolution[C]// 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 5791-5800.
12 ZENG Y, FU J, CHAO H, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]// 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Long Bench: IEEE, 2019: 1486-1494.
13 YI Z, TANG Q, AZIZI S, et al. Contextual residual aggregation for ultra high-resolution image inpainting[C]// 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 7508-7517.
14 PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 2536-2544.
15 SATOSHI I, EDGAR S, HIROSHI I Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36 (4): 107
16 YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City: IEEE, 2018: 5505-5514.
17 GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of wasserstein GANs[C]// Thirty-first Conference on Neural Information Processing Systems. Long Bench: NIPS, 2017: 5767-5677.
[1] 赫贵然,李奇,冯华君,徐之海,陈跃庭. 基于CNN特征提取的双焦相机连续数字变焦[J]. 浙江大学学报(工学版), 2019, 53(6): 1182-1189.