Image quality assessment with deep pooling of visual feature

doi:10.3785/j.issn.1008-973X.2019.03.012

Journal of ZheJiang University (Engineering Science)

2019, Vol. 53

Issue (3): 512-521 DOI: 10.3785/j.issn.1008-973X.2019.03.012

Computer Technology

Image quality assessment with deep pooling of visual feature

Ming-kun FENG(

),Xiang SHI

School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China

Download:

HTML

PDF(890KB) HTML
Export: BibTeX | EndNote (RIS)

Abstract

The neural network pooling prediction model of visual multi-channel was constructed and an image quality assessment method based on deep pooling of visual feature was proposed, aiming at the shortcoming of current research on visual perception characteristic and image feature assessment algorithms. Firstly, two complementary visual assessment algorithms based on histogram statistics and singular value decomposition were designed with human visual system characteristics. Further, the sparse gradient information of every visual channel for one image was deeply processed. Secondly, the multi-channel assessment pooling of every visual feature layer was predicted, respectively, by constructing a pooling model of the BP neural network. Finally, the visual feature assessment of three layers was deep adaptively pooled from the inner layer to the outer layer. The experiment results show that the constructed pooling model effectively improves the index level of every assessment algorithm, and the proposed method achieves great advantage compared to the existing methods.

Key words： image quality assessment BP neural network deep feature processing visual perception characteristic pooling prediction model

Received: 16 July 2018 Published: 04 March 2019

CLC:

TH 741.1

	Service
	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	Ming-kun FENG
	Xiang SHI

Cite this article:

Ming-kun FENG,Xiang SHI. Image quality assessment with deep pooling of visual feature. Journal of ZheJiang University (Engineering Science), 2019, 53(3): 512-521.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2019.03.012 OR http://www.zjujournals.com/eng/Y2019/V53/I3/512

视觉特征深度融合的图像质量评价

针对当前视觉感知特性研究和图像特征评价算法的不足，通过构建视觉多通道神经网络融合预测模型，提出一种视觉特征深度融合的图像质量评价方法. 首先，结合人类视觉系统特性设计直方图统计和奇异值分解2个互补视觉评价算法，进一步对图像各视觉通道的稀疏化梯度信息进行深度处理. 其次，构建BP神经网络融合模型，对各层视觉特征的多通道评价融合分别进行预测. 最后，对3层视觉特征评价从内层到外层逐层地进行深度自适应融合. 实验结果表明，所构建的融合模型有效提高了各种评价算法的指标水平，所提方法优于已有方法.

关键词： 图像质量评价, BP神经网络, 深度特征处理, 视觉感知特性, 融合预测模型

Fig.1 Schematic diagram of deep perception processing and neural network prediction （DPPNNP） method with fusion of image features

失真类型参数设置	指标	MSSIM	IFC	VIF	GSM	FSIM	VSI	SVD	IGM	MAD	DPPNNP
JPEG2000 $\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{4}}}&{{\gamma _{{\rm{12}}}} = 0.{0}1} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.02}}}&{{\gamma _{{\rm{22}}}} = 0.0{01}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{00}}1} \end{array}$	RMSE	4.414 2	4.696 1	4.714 8	4.179 3	4.408 6	5.496 8	4.445 8	3.310 5	3.059 1	3.873 1
	PLCC	0.983 3	0.981 1	0.980 9	0.985 1	0.983 4	0.974 0	0.983 1	0.990 7	0.992 0	0.987 2
	SROCC	0.984 5	0.964 1	0.978 7	0.986 9	0.988 4	0.969 8	0.981 4	0.986 6	0.987 1	0.984 1
JPEG $\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{3}}}&{{\gamma _{{\rm{12}}}} = 0.1} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.02}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.001}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{10}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{4}}} \end{array}$	RMSE	3.600 4	3.251 14	2.802 7	3.909 6	3.861 5	3.941 4	3.419 4	2.336 6	4.532 8	1.936 9
	PLCC	0.989 3	0.991 3	0.993 6	0.987 4	0.987 7	0.987 2	0.990 4	0.995 5	0.983 0	0.996 9
	SROCC	0.977 9	0.953 8	0.965 0	0.965 0	0.980 1	0.954 1	0.953 6	0.976 0	0.967 1	0.975 8
WN $\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = {0}{\rm{.4}}}&{{\gamma _{{\rm{12}}}} = 0.{01}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.04}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.01}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = {0}{\rm{.000\;1}}} \end{array}$	RMSE	2.866 7	3.078 6	4.131 3	3.307 9	3.933 6	3.335 2	2.374 2	1.659 0	3.124 3	1.260 6
	PLCC	0.992 8	0.991 7	0.985 1	0.990 5	0.986 5	0.990 3	0.995 1	0.997 6	0.991 5	0.998 6
	SROCC	0.984 7	0.957 7	0.966 6	0.987 9	0.976 9	0.974 7	0.994 0	0.994 5	0.991 3	0.993 3
Gblur $\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = {0}{\rm{.4}}}&{{\gamma _{{\rm{12}}}} = 0.{01}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.04}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.01}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = {0}{\rm{.000\;1}}} \end{array}$	RMSE	5.454 9	3.859 5	7.503 8	5.505 4	4.566 3	5.523 8	6.776 2	3.629 6	3.893 0	3.135 4
	PLCC	0.976 3	0.988 2	0.954 6	0.975 8	0.983 4	0.975 7	0.963 1	0.989 6	0.988 0	0.992 2
	SROCC	0.989 2	0.969 0	0.933 4	0.987 9	0.991 6	0.967 9	0.963 5	0.988 8	0.987 7	0.987 7
Fastfading $\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{3}}}&{{\gamma _{{\rm{12}}}} = 0.{\rm{1}}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.02}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.001}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{00}}1} \end{array}$	RMSE	4.818 3	5.117 4	4.125 8	5.078 8	5.258 1	5.553 9	6.184 3	5.086 5	4.192 0	2.838 3
	PLCC	0.979 7	0.977 1	0.985 2	0.977 5	0.975 8	0.973 0	0.966 4	0.977 4	0.984 7	0.993 6
	SROCC	0.984 1	0.957 0	0.969 5	0.982 7	0.981 5	0.964 4	0.969 9	0.972 0	0.985 0	0.990 7
失真类型总和 $\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = {0}{\rm{.28}}}&{{\gamma _{{\rm{12}}}} = 0.{\rm{3}}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.4}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.001}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{6}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{08}}} \end{array}$	RMSE	6.231 9	5.371 8	7.469 2	6.103 1	5.784 3	6.597 9	6.279 0	4.722 0	5.517 2	4.132 4
	PLCC	0.967 3	0.975 8	0.952 6	0.968 6	0.971 9	0.963 2	0.966 8	00.981 3	0.974 4	0.987 5
	SROCC	0.965 1	0.955 2	0.920 6	0.971 8	0.977 8	0.949 1	0.967 2	0.974 1	0.977 3	0.981 5

Tab.2 Assessment results comparison between proposed DPPNNP and other methods on database LIVE

Fig.2 Predicting model of multi-channel assessment pooling based on BP neural network

Tab.1 Results comparison of BP neural network predicting vs CSF pooling

Tab.3 Assessment results comparison between proposed DPPNNP and other methods on databases CSIQ and tid2013

Fig.3 Scatter distribution plot of various methods

Tab.4 Test results comparisons of time complexity for different image quality assessment methods


[1]	GUO Y C, HAO Y T, YU M Image retargeting quality assessment based on content deformation measurement[J]. Signal Processing: Image Communication, 2018, 67 (6): 171- 181

[2]	CHEN Z B, LIN J X, LIAO N, et al Full reference quality assessment for image retargeting based on natural scene statistics modeling and bi-directional saliency similarity[J]. IEEE Transactions on Image Processing, 2017, 26 (11): 5138- 5148 doi: 10.1109/TIP.2017.2736422

[3]	ZHANG Y CH, KING N N, MA L, et al Objective quality assessment of image retargeting by incorporating fidelity measures and inconsistency detection[J]. IEEE Transactions on Image Processing, 2017, 26 (11): 5980- 5993

[4]	ZHANG Y B, LIN W S, LI Q H, et al Multiple-level feature-based measure for retargeted image quality[J]. IEEE Transactions on Image Processing, 2018, 27 (1): 451- 463 doi: 10.1109/TIP.2017.2761556

[5]	丰明坤, 王中鹏, 叶绿视觉稀疏化多通道多特征自适应的图像评价[J]. 仪器仪表学报, 2016, 37 (3): 667- 674 FENG Ming-kun, WANG ZHong-peng, YE Lv Image quality assessment based on adaptive sparse visual multi-channel and multi-feature pooling[J]. Chinese Journal of Scientific Instrument, 2016, 37 (3): 667- 674 doi: 10.3969/j.issn.0254-3087.2016.03.025

[6]	BAMPIS C G, GUPTA P, SOUNDARARAJAN R, et al SpEED-QA: patial efficient entropic differencing for image and video quality[J]. IEEE Signal Processing Letters, 2017, 24 (9): 1333- 1337 doi: 10.1109/LSP.2017.2726542

[7]	NI ZH K, MA L, ZENG H Q, et al Gradient direction for screen content image quality assessment[J]. IEEE Signal Processing Letters, 2016, 23 (10): 1394- 1398 doi: 10.1109/LSP.2016.2599294

[8]	KIM J, ZENG H, GHADIYARAM D, et al Deep convolutional neural models for picture-quality prediction: challenges and solutions to data-driven image quality assessment[J]. IEEE Signal Processing Magazine, 2017, 34 (6): 130- 141 doi: 10.1109/MSP.2017.2736018

[9]	GHADIYARAM D, BOVIK A. C Massive online crowdsourced study of subjective and objective picture quality[J]. IEEE Transactions on Image Process, 2016, 25 (1): 372- 387 doi: 10.1109/TIP.2015.2500021

[10]	LARSON E C, CHANDLER D M Most apparent distortion: full-reference image quality assessment and the role of strategy[J]. Journal of Electronic Imaging, 2010, 19 (1): 011006-1- 011006-21

[11]	ZHANG L, ZHANG L, MOU X Q, et al FSIM: a feature similarity index for image quality assessment[J]. IEEE Transactions on Image Processing, 2011, 20 (8): 2378- 2386 doi: 10.1109/TIP.2011.2109730

[12]	ZHANG L, SHEN Y LI H Y VSI: a visual saliency-induced index for perceptual image quality assessment[J]. IEEE Transactions on Image Processing, 2014, 23 (10): 4270- 4281 doi: 10.1109/TIP.2014.2346028

[13]	DING Y, WANG SH Z, ZHANG D Full-reference image quality assessment using statistical local correlation[J]. Electronics Letters, 2014, 50 (2): 79- 81 doi: 10.1049/el.2013.3365

[14]	WU J J, LIN W S, SHI G M Perceptual quality metric with internal generative mechanism[J]. IEEE Transactions on Image Processing, 2013, 22 (1): 43- 54 doi: 10.1109/TIP.2012.2214048

[15]	丰明坤, 赵生妹, 邢超基于视觉显著失真度的图像质量自适应评价方法[J]. 电子与信息学报, 2015, 37 (9): 2062- 2068 FENG Ming-kun, ZHAO SHeng-mei, XING CHao Image quality self-adaptive assessment based on visual salience distortion[J]. Journal of Electronics & Information Technology, 2015, 37 (9): 2062- 2068

[16]	丰明坤, 赵生妹, 施祥视觉多通道梯度与低阶矩自适应图像评价[J]. 仪器仪表学报, 2015, 36 (11): 2531- 2537 FENG Ming-kun, ZHAO SHeng-mei, SHI Xiang Adaptive image quality assessment based on visual multi-channel gradient and low order moment[J]. Chinese Journal of Scientific Instrument, 2015, 36 (11): 2531- 2537 doi: 10.3969/j.issn.0254-3087.2015.11.017

[17]	WANG ZH, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612 doi: 10.1109/TIP.2003.819861

[18]	WANG ZH, LU L G, BOVIK A C Foveation scalable video coding with automatic fixation selection[J]. IEEE Transactions on Image Processing, 2003, 12 (2): 243- 254 doi: 10.1109/TIP.2003.809015

[19]	LIU A M, LIN W S, NARWARIA M Image quality assessment based on gradient similarity[J]. IEEE Transaction on Image Processing, 2012, 21 (4): 1500- 1512 doi: 10.1109/TIP.2011.2175935

[20]	GAO X B, LU W, TAO D CH, et al Image quality assessment based on multiscale geometric analysis[J]. IEEE Transactions on Image Processing, 2009, 18 (7): 1409- 1423 doi: 10.1109/TIP.2009.2018014

[21]	林志洁, 丰明坤深度视觉特征与策略互补融合的图像质量评价[J]. 模式识别与人工智能, 2017, 30 (8): 682- 691 LIN ZHi-jie, FENG Ming-kun Image quality assessment based on complementary pooling of deeply visual feature and strategy[J]. Pattern Recognition and Artificial Intelligence, 2017, 30 (8): 682- 691

[22]	SHNAYDERMAN A, GUSEV A, ESKICIOGLU A M An SVD-based grayscale image quality measure for local and global assessment[J]. IEEE Transactions on Image Processing, 2006, 15 (2): 422- 429 doi: 10.1109/TIP.2005.860605

[23]	HU A ZH, ZHANG R YIN D, et al Image quality assessment using an SVD-based structural projection[J]. Signal Processing: Image Communication, 2014, 29 (3): 293- 302 doi: 10.1016/j.image.2014.01.007

[24]	WANG ZH, SIMONCELLI E P, BOVIK A C. Multi-scale structural similarity for image quality assessment [C]// Proceedings of the 37th IEEE Asilomar Conference on Signals, Systems and Computers. Pacific Grove: IEEE, 2002(2): 1398–1402.

[25]	SHEIKH H R, BOVIK A C, VECIANA G D An information fidelity criterion for image quality assessment using natural scene statistics[J]. IEEE Transactions on Image Processing, 2005, 14 (12): 2117- 2128 doi: 10.1109/TIP.2005.859389

[1]	Xu-ming SU,Cheng-gang FANG,Yu-bin PAN,Wei-wei WU,Ya-ping LI,Lang ZHU. Modeling error of visual measurement system under changing illuminance[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1929-1935.

[2]	XIE Jie, HUANG Xiao-diao, FANG Cheng-gang, ZHOU Bao-cang, LU Ning. Thermal characteristics and thermal error modeling analysis for motorized spindle of gear grinding machine tool[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(2): 247-254.

[3]	MENG Jun, DENG Xiao-yu, YU Jie-zhou. Postoperative survival prediction model of BP neural network with variable cluster[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(12): 2365-2371.

[4]	WU Jiang hong, XUE Zhi qiang, JIN Peng, LI Hui xi. Temperature maldistribution in micro-channel heat exchanger applied to electrical vehicle’s heat pump air conditioning[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(8): 1537-1544.

[5]	LIU Lei, YANG Peng,LIU Zuo-jun1. Lower limb locomotion-mode identification based on multi-source information and particle swarm optimization algorithm[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(3): 439-447.

[6]	HUANG Xiao-shuo,HE Yan,JIANG Jing-ping. Internet based control strategy for brushless DC motor drive systems [J]. Journal of ZheJiang University (Engineering Science), 2013, 47(5): 831-836.

[7]	WANG Xiang, DING Yong. Full reference image quality assessment based on Gabor filter[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(3): 422-430.

[8]	TONG Shui-guang, WANG Xiang-bing, ZHONG Wei, ZHANG Jian. Dynamic optimization design for rigid landing leg of crane based on BP-HGA[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(1): 122-130.

[9]	WANG De-ming, WANG Li,ZHANG Guang-ming. Short-term wind speed forecast model for wind farms based on genetic BP neural network[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(5): 837-841.

[10]	ZHAO Zhen, ZHANG Shu-you. Hybrid current model of breaking cycle and its application[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(2): 301-308.

[11]	LOU Bin, CHEN Hai-Bin, DIAO Wu-Feng, et al. No-reference image quality assessment based on statistical model of natural image[J]. Journal of ZheJiang University (Engineering Science), 2010, 44(2): 248-252.

[12]	LOU Bin, CHEN Hai-Bin, DIAO Wu-Feng, et al. Structural similarity image quality assessment based on distortion model[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(5): 864-868.

[13]	XU Jing-hua, ZHANG Shu-you. Shape retrieval method of 3D models based on shape distribution graph and BP neural network[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(5): 877-883.

Viewed

Full text

Abstract

Cited

Shared

Discussed