Dual-focal camera continuous digital zoom based onCNN and feature extraction

doi:10.3785/j.issn.1008-973X.2019.06.018

Journal of ZheJiang University (Engineering Science)

2019, Vol. 53

Issue (6): 1182-1189 DOI: 10.3785/j.issn.1008-973X.2019.06.018

Computer and Aut omation Technology

Dual-focal camera continuous digital zoom based onCNN and feature extraction

Gui-ran HE(

),Qi LI*(

),Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN

College of Science and Engineering, Zhejiang University, Hangzhou 310058, China

Download:

HTML

PDF(1210KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

A digital continuous zooming scheme mainly for smartphone dual-focal lens was proposed, which was based on extraction and patch-matching of the convolutional neural network feature map, making full use of the shooting information of two different focal length lenses. The high-resolution information of the long-focal image was migrated to the repairable area of the short-focus image, then the zoomed image was produced accordingly. The simulation and experimental results show that the image produced by the proposed method has higher subjective resolution and better visual clarity, compared with the traditional interpolation and super-resolution results. The method can greatly enhance the imaging quality of the dual-focal lens when the magnification factor is between the two lens. For the texture outside the Short-Focal-Len filed but in the Long-Focal-Len field, the proposed method also performs better than the existing methods. Otherwise, the proposed method has good robustness in dealing with the parallax problem of the dual-focal camera.

Key words： asymmetric dual-focal camera continuous digital zoom convolutional neural network (CNN) feature extraction super-resolution image restoration

Received: 24 October 2018 Published: 22 May 2019

CLC:

TP 751.1

Corresponding Authors: Qi LI E-mail: 3130103692@zju.edu.cn;liqi@zju.edu.cn

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Gui-ran HE
	Qi LI
	Hua-jun FENG
	Zhi-hai XU
	Yue-ting CHEN

Cite this article:

Gui-ran HE,Qi LI,Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN. Dual-focal camera continuous digital zoom based onCNN and feature extraction. Journal of ZheJiang University (Engineering Science), 2019, 53(6): 1182-1189.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2019.06.018 OR http://www.zjujournals.com/eng/Y2019/V53/I6/1182

基于CNN特征提取的双焦相机连续数字变焦

设计一种主要针对智能手机双焦镜头的连续数字变焦方案. 该方案基于卷积神经网络特征层提取和特征块匹配，充分利用2个不同焦距镜头的拍摄信息，将长焦镜头图像的高分辨信息迁移到短焦图像的可修复区域，并以修复后的短焦图像为基础进行数字变焦. 仿真和实验结果表明，相比基于传统插值放大的变焦方案和基于单图像超分辨的变焦方案，所提方案的处理结果拥有更高的主观分辨率和视觉清晰度；当用户给定的变焦倍数在长、短焦镜头倍数之间时，所提方案可以显著提升变焦图像的质量；对于处于长焦相机视场外、短焦相机视场内的纹理，修复效果比现有方法更好；该方法的处理结果对长焦、短焦图像的双目视差大小有着很好的鲁棒性.

关键词： 非对称双焦镜头, 连续数字变焦, 卷积神经网络（CNN）, 特征提取, 超分辨成像, 图像恢复

Fig.1 Texture restoring results of stimulated images with PatchMatch method

Fig.2 Schematic diagram of classical asymmetric dual zooming imaging model

Fig.3 Scenery images used for stimulation experiments

Fig.4 Comparison of stimulation results for proposed texture restoring method and other methods

Fig.5 Comparison of zooming results for different zooming scales using different methods

Tab.1 Quantitative evaluation of zooming results in stimulation experiments

Fig.6 Texture restoring examination for field out of Long-Focal-Len

Fig.7 Experimental result of real image and verification of parallax problem


[1]	YANG J, WRIGHT J, HUANG T S, et al Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19 (11): 2861- 73 doi: 10.1109/TIP.2010.2050625

[2]	TIMOFTE R, SMET V D, GOOL L V. A+: Adjusted anchored neighborhood regression for fast super-resolution [C] // Asian Conference on Computer Vision 2014. Singapore: Springer, 2014: 111-126.

[3]	TIMOFTE R, DE V, VAN GOOL L. Anchored neighborhod regression for fast example-based super-resolution [C] // Proceedings of the IEEE International Conference on Computer Vision 2013. Sydney: ICCV, 2013: 1920-1927.

[4]	ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations [C] // International Conference on Curves and Surfaces 2010. Avignon: Springer, 2010: 711-730.

[5]	DONG C, LOY C, HE K, et al Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (2): 295- 307 doi: 10.1109/TPAMI.2015.2439281

[6]	KIM J, LEE JK, LEE KM. Accurate image super-resolution using very deep convolutional networks [C] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: [s. n.], 2016: 1646-1654.

[7]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets [C] // Advances in Neural Information Processing Systems. Montreal: [s. n.], 2014: 2672-2680.

[8]	YU S, MOON B, KIM D, et al Continuous digital zooming of asymmetric dual camera images using registration and variational image restoration[J]. Multidimensional Systems and Signal Processing, 2018, 29 (4): 1959- 87 doi: 10.1007/s11045-017-0534-4

[9]	MOON B, YU S, KO S, et al Continuous digital zooming using local self-similarity-based super-resolution for an asymmetric dual camera system[J]. Journal of the Optical Society of America A-optics Image Science and Vision, 2017, 34 (6): 991- 1003 doi: 10.1364/JOSAA.34.000991

[10]	MA H, LI Q, XU Z, et al Photo-realistic continuous digital zooming for an asymmetrical dual camera system[J]. Optics and Laser Technology, 2019, 109: 110- 22 doi: 10.1016/j.optlastec.2018.07.056

[11]	CRIMINISI A, PEREZ P, TOYAMA K Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13 (9): 1200- 12 doi: 10.1109/TIP.2004.833105

[12]	CHEN H, CHEN C, CHUANG Y. A deep convolutional neural network for continuous zoom with dual cameras [C] // SIGGRAPH Asia 2017 Posters. Bangkok: ACM, 2017: 13.

[13]	LI C, WAND M. Combining markov random fields and convolutional neural networks for image synthesis [C] // Conference on Computer Vision and Pattern Recognition (CVPR) 2016. Las Vegas: [s. n.], 2016: 2479-2486.

[14]	EFROS A, LEUNG T K. Texture synthesis by non-parametric sampling [C] // Proceedings of the Seventh IEEE International Conference on Computer Vision 1999. Kerkyra: IEEE, 1999: 1033-1038.

[1]	Fang LIU,Zhen WANG,Rui-di LIU,Kai WANG. Short-term forecasting method of wind power generation based on BP neural network with combined loss function[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 594-600.

[2]	Qiao-hong CHEN,YI CHEN,Wen-shu Li,Yu-bo JIA. Clothing image classification based on multi-scale SE-Xception[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1727-1735.

[3]	Ping YANG,Dan WANG,Zi-jian KAGN,Tong LI,Li-hua FU,Yue-ren YU. Prediction model of paroxysmal atrial fibrillation based on pattern recognition and ensemble CNN-LSTM[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(5): 1039-1048.

[4]	Yi-xiong FENG,Kang-jie LI,Yi-cong GAO,Hao ZHEN,Jian-rong TAN. Shaft surface defect detection method based on feature and morphology reconstruction[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 427-434.

[5]	Mei-ying QIAO,Xia-xia TANG,Shu-hao YAN,Jian-ke SHI. Bearing fault diagnosis based on improved sparse filter and deep network fusion[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(12): 2301-2309.

[6]	Zi-yu JIA,You-fang LIN,Hong-jun ZHANG,Jing WANG. Sleep stage classification model based ondeep convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1899-1905.

[7]	Hong-guang LI,Ying GUO,Ping SUI,Zi-sen QI. Frequency hopping modulation recognition of convolutional neural network based on time-frequency characteristics[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1945-1954.

[8]	Gang YE,Yi-bo LI,Zhu-xi MA,Jie CHENG. End-to-end aluminum strip surface defects detection and recognition method based on ViBe[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1906-1914.

[9]	Xiao-hu ZHAO,Liang-fei YIN,Cheng-long ZHAO. Image captioning based on global-local feature and adaptive-attention[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(1): 126-134.

[10]	Ran DUAN,Deng-wen ZHOU,Li-juan ZHAO,Xiao-liang CHAI. Image super-resolution reconstruction based on multi-scale feature mapping network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(7): 1331-1339.

[11]	Yan LV,Meng ZHANG,Wu-hao JIANG,Yi-hua NI,Xiao-hong QIAN. Design of elderly fall detection system using CNN[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(6): 1130-1138.

[12]	Yue DONG,Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN,Qi LI. Attention Res-Unet: an efficient shadow detection algorithm[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(2): 373-381.

[13]	Jing-chang WANG,Ling CHEN,Shan-shan YU,Chen-shu JIANG,Yong WU. Multi-factor perceived short-term tourist number prediction model based on gated recurrent unit[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(12): 2357-2364.

[14]	Xiang-long ZHAO,Jie CHEN,Rong-jing HONG,Hua WANG,Yuan-yuan LI. Adaptive feature extraction method for slewing bearing based on Wavelet leader and optimized isometric mapping method[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(11): 2092-2101.

[15]	Hao LI,Wen-jie ZHAO,Bo HAN. Convolutional neural network acceleration algorithm based on filters pruning[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(10): 1994-2002.

Viewed

Full text

Abstract

Cited

Shared

Discussed