Please wait a minute...
浙江大学学报(工学版)  2019, Vol. 53 Issue (6): 1182-1189    DOI: 10.3785/j.issn.1008-973X.2019.06.018
计算机与自动化技术     
基于CNN特征提取的双焦相机连续数字变焦
赫贵然(),李奇*(),冯华君,徐之海,陈跃庭
浙江大学 光电科学与工程学院,浙江 杭州 310058
Dual-focal camera continuous digital zoom based onCNN and feature extraction
Gui-ran HE(),Qi LI*(),Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN
College of Science and Engineering, Zhejiang University, Hangzhou 310058, China
 全文: PDF(1210 KB)   HTML
摘要:

设计一种主要针对智能手机双焦镜头的连续数字变焦方案. 该方案基于卷积神经网络特征层提取和特征块匹配,充分利用2个不同焦距镜头的拍摄信息,将长焦镜头图像的高分辨信息迁移到短焦图像的可修复区域,并以修复后的短焦图像为基础进行数字变焦. 仿真和实验结果表明,相比基于传统插值放大的变焦方案和基于单图像超分辨的变焦方案,所提方案的处理结果拥有更高的主观分辨率和视觉清晰度;当用户给定的变焦倍数在长、短焦镜头倍数之间时,所提方案可以显著提升变焦图像的质量;对于处于长焦相机视场外、短焦相机视场内的纹理,修复效果比现有方法更好;该方法的处理结果对长焦、短焦图像的双目视差大小有着很好的鲁棒性.

关键词: 非对称双焦镜头连续数字变焦卷积神经网络(CNN)特征提取超分辨成像图像恢复    
Abstract:

A digital continuous zooming scheme mainly for smartphone dual-focal lens was proposed, which was based on extraction and patch-matching of the convolutional neural network feature map, making full use of the shooting information of two different focal length lenses. The high-resolution information of the long-focal image was migrated to the repairable area of the short-focus image, then the zoomed image was produced accordingly. The simulation and experimental results show that the image produced by the proposed method has higher subjective resolution and better visual clarity, compared with the traditional interpolation and super-resolution results. The method can greatly enhance the imaging quality of the dual-focal lens when the magnification factor is between the two lens. For the texture outside the Short-Focal-Len filed but in the Long-Focal-Len field, the proposed method also performs better than the existing methods. Otherwise, the proposed method has good robustness in dealing with the parallax problem of the dual-focal camera.

Key words: asymmetric dual-focal camera    continuous digital zoom    convolutional neural network (CNN)    feature extraction    super-resolution    image restoration
收稿日期: 2018-10-24 出版日期: 2019-05-22
CLC:  TP 751.1  
通讯作者: 李奇     E-mail: 3130103692@zju.edu.cn;liqi@zju.edu.cn
作者简介: 赫贵然(1995—),男,硕士生,从事数字图像处理研究. orcid.org/0000-0003-3352-2991. E-mail: 3130103692@zju.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
赫贵然
李奇
冯华君
徐之海
陈跃庭

引用本文:

赫贵然,李奇,冯华君,徐之海,陈跃庭. 基于CNN特征提取的双焦相机连续数字变焦[J]. 浙江大学学报(工学版), 2019, 53(6): 1182-1189.

Gui-ran HE,Qi LI,Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN. Dual-focal camera continuous digital zoom based onCNN and feature extraction. Journal of ZheJiang University (Engineering Science), 2019, 53(6): 1182-1189.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2019.06.018        http://www.zjujournals.com/eng/CN/Y2019/V53/I6/1182

图 1  应用块匹配(PatchMatch)对仿真图像进行纹理修复
图 2  非对称双焦摄像头典型成像模型示意图
图 3  用于仿真实验的自然景物图像
图 4  本文细节修复方法与其他方法的仿真结果对比
图 5  不同变焦倍数、采用不同方法的变焦结果对比
图像 M =1.3 M =1.5 M =1.7
双线性插值 A+ 本文方法 双线性插值 A+ 本文方法 双线性插值 A+ 本文方法
Ancient 27.45 29.81 27.78 24.32 25.52 24.46 25.62 27.46 27.30
Autumn 25.27 25.93 26.65 21.59 22.03 22.28 23.60 24.27 25.52
Baboon 27.20 28.57 27.54 23.53 24.27 23.62 27.47 28.09 28.16
Building1 27.20 29.33 28.17 23.13 23.93 23.67 24.27 25.57 26.01
Building2 29.26 31.79 29.48 25.66 26.71 25.87 27.45 29.60 29.23
Building3 23.33 25.57 24.89 20.08 21.00 20.95 20.21 21.59 22.19
Butterfly 29.73 34.79 31.23 28.77 31.97 29.97 26.96 31.27 29.54
Fruits 29.32 31.62 30.85 27.12 28.56 28.02 27.38 29.38 30.11
Shanghai 26.07 28.32 27.26 22.21 22.99 22.83 22.88 24.02 24.24
Mountain 29.75 30.80 31.31 26.04 26.43 26.77 27.43 28.17 29.64
表 1  仿真实验变焦结果定量评价
图 6  长焦视场外纹理修复效果验证实验
图 7  真实图像实验结果及视差鲁棒性验证
1 YANG J, WRIGHT J, HUANG T S, et al Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19 (11): 2861- 73
doi: 10.1109/TIP.2010.2050625
2 TIMOFTE R, SMET V D, GOOL L V. A+: Adjusted anchored neighborhood regression for fast super-resolution [C] // Asian Conference on Computer Vision 2014. Singapore: Springer, 2014: 111-126.
3 TIMOFTE R, DE V, VAN GOOL L. Anchored neighborhod regression for fast example-based super-resolution [C] // Proceedings of the IEEE International Conference on Computer Vision 2013. Sydney: ICCV, 2013: 1920-1927.
4 ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations [C] // International Conference on Curves and Surfaces 2010. Avignon: Springer, 2010: 711-730.
5 DONG C, LOY C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (2): 295- 307
doi: 10.1109/TPAMI.2015.2439281
6 KIM J, LEE JK, LEE KM. Accurate image super-resolution using very deep convolutional networks [C] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: [s. n.], 2016: 1646-1654.
7 GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets [C] // Advances in Neural Information Processing Systems. Montreal: [s. n.], 2014: 2672-2680.
8 YU S, MOON B, KIM D, et al Continuous digital zooming of asymmetric dual camera images using registration and variational image restoration[J]. Multidimensional Systems and Signal Processing, 2018, 29 (4): 1959- 87
doi: 10.1007/s11045-017-0534-4
9 MOON B, YU S, KO S, et al Continuous digital zooming using local self-similarity-based super-resolution for an asymmetric dual camera system[J]. Journal of the Optical Society of America A-optics Image Science and Vision, 2017, 34 (6): 991- 1003
doi: 10.1364/JOSAA.34.000991
10 MA H, LI Q, XU Z, et al Photo-realistic continuous digital zooming for an asymmetrical dual camera system[J]. Optics and Laser Technology, 2019, 109: 110- 22
doi: 10.1016/j.optlastec.2018.07.056
11 CRIMINISI A, PEREZ P, TOYAMA K Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13 (9): 1200- 12
doi: 10.1109/TIP.2004.833105
12 CHEN H, CHEN C, CHUANG Y. A deep convolutional neural network for continuous zoom with dual cameras [C] // SIGGRAPH Asia 2017 Posters. Bangkok: ACM, 2017: 13.
13 LI C, WAND M. Combining markov random fields and convolutional neural networks for image synthesis [C] // Conference on Computer Vision and Pattern Recognition (CVPR) 2016. Las Vegas: [s. n.], 2016: 2479-2486.
14 EFROS A, LEUNG T K. Texture synthesis by non-parametric sampling [C] // Proceedings of the Seventh IEEE International Conference on Computer Vision 1999. Kerkyra: IEEE, 1999: 1033-1038.
[1] 刘芳,汪震,刘睿迪,王锴. 基于组合损失函数的BP神经网络风力发电短期预测方法[J]. 浙江大学学报(工学版), 2021, 55(3): 594-600.
[2] 陈巧红,陈翊,李文书,贾宇波. 多尺度SE-Xception服装图像分类[J]. 浙江大学学报(工学版), 2020, 54(9): 1727-1735.
[3] 张彦楠,黄小红,马严,丛群. 基于深度学习的录音文本分类方法[J]. 浙江大学学报(工学版), 2020, 54(7): 1264-1271.
[4] 杨萍,王丹,康子健,李童,付利华,余悦任. 基于模式识别和集成CNN-LSTM的阵发性房颤预测模型[J]. 浙江大学学报(工学版), 2020, 54(5): 1039-1048.
[5] 冯毅雄,李康杰,高一聪,郑浩,谭建荣. 基于特征与形貌重构的轴件表面缺陷检测方法[J]. 浙江大学学报(工学版), 2020, 54(3): 427-434.
[6] 乔美英,汤夏夏,闫书豪,史建柯. 基于改进稀疏滤波与深度网络融合的轴承故障诊断[J]. 浙江大学学报(工学版), 2020, 54(12): 2301-2309.
[7] 贾子钰,林友芳,张宏钧,王晶. 基于深度卷积神经网络的睡眠分期模型[J]. 浙江大学学报(工学版), 2020, 54(10): 1899-1905.
[8] 李红光,郭英,眭萍,齐子森. 基于时频特征的卷积神经网络跳频调制识别[J]. 浙江大学学报(工学版), 2020, 54(10): 1945-1954.
[9] 叶刚,李毅波,马逐曦,成杰. 基于ViBe的端到端铝带表面缺陷检测识别方法[J]. 浙江大学学报(工学版), 2020, 54(10): 1906-1914.
[10] 赵小虎,尹良飞,赵成龙. 基于全局?局部特征和自适应注意力机制的图像语义描述算法[J]. 浙江大学学报(工学版), 2020, 54(1): 126-134.
[11] 吕艳,张萌,姜吴昊,倪益华,钱小鸿. 采用卷积神经网络的老年人跌倒检测系统设计[J]. 浙江大学学报(工学版), 2019, 53(6): 1130-1138.
[12] 董月,冯华君,徐之海,陈跃庭,李奇. Attention Res-Unet: 一种高效阴影检测算法[J]. 浙江大学学报(工学版), 2019, 53(2): 373-381.
[13] 王敬昌,陈岭,余珊珊,蒋晨书,吴勇. 基于门控循环单元的多因素感知短期游客人数预测模型[J]. 浙江大学学报(工学版), 2019, 53(12): 2357-2364.
[14] 赵祥龙,陈捷,洪荣晶,王华,李媛媛. 基于Wavelet leader和优化的等距映射算法的回转支承自适应特征提取[J]. 浙江大学学报(工学版), 2019, 53(11): 2092-2101.
[15] 李浩,赵文杰,韩波. 基于滤波器裁剪的卷积神经网络加速算法[J]. 浙江大学学报(工学版), 2019, 53(10): 1994-2002.