Please wait a minute...
浙江大学学报(工学版)  2019, Vol. 53 Issue (3): 512-521    DOI: 10.3785/j.issn.1008-973X.2019.03.012
计算机技术     
视觉特征深度融合的图像质量评价
丰明坤(),施祥
浙江科技学院 信息与电子工程学院,浙江 杭州 310023
Image quality assessment with deep pooling of visual feature
Ming-kun FENG(),Xiang SHI
School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023, China
 全文: PDF(890 KB)   HTML
摘要:

针对当前视觉感知特性研究和图像特征评价算法的不足,通过构建视觉多通道神经网络融合预测模型,提出一种视觉特征深度融合的图像质量评价方法. 首先,结合人类视觉系统特性设计直方图统计和奇异值分解2个互补视觉评价算法,进一步对图像各视觉通道的稀疏化梯度信息进行深度处理. 其次,构建BP神经网络融合模型,对各层视觉特征的多通道评价融合分别进行预测. 最后,对3层视觉特征评价从内层到外层逐层地进行深度自适应融合. 实验结果表明,所构建的融合模型有效提高了各种评价算法的指标水平,所提方法优于已有方法.

关键词: 图像质量评价BP神经网络深度特征处理视觉感知特性融合预测模型    
Abstract:

The neural network pooling prediction model of visual multi-channel was constructed and an image quality assessment method based on deep pooling of visual feature was proposed, aiming at the shortcoming of current research on visual perception characteristic and image feature assessment algorithms. Firstly, two complementary visual assessment algorithms based on histogram statistics and singular value decomposition were designed with human visual system characteristics. Further, the sparse gradient information of every visual channel for one image was deeply processed. Secondly, the multi-channel assessment pooling of every visual feature layer was predicted, respectively, by constructing a pooling model of the BP neural network. Finally, the visual feature assessment of three layers was deep adaptively pooled from the inner layer to the outer layer. The experiment results show that the constructed pooling model effectively improves the index level of every assessment algorithm, and the proposed method achieves great advantage compared to the existing methods.

Key words: image quality assessment    BP neural network    deep feature processing    visual perception characteristic    pooling prediction model
收稿日期: 2018-07-16 出版日期: 2019-03-04
CLC:  TH 741.1  
作者简介: 丰明坤(1978—),男,讲师,从事信息智能感知处理研究. orcid.org/0000-0001-6716-8949. E-mail: fmk_78_62@163.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
丰明坤
施祥

引用本文:

丰明坤,施祥. 视觉特征深度融合的图像质量评价[J]. 浙江大学学报(工学版), 2019, 53(3): 512-521.

Ming-kun FENG,Xiang SHI. Image quality assessment with deep pooling of visual feature. Journal of ZheJiang University (Engineering Science), 2019, 53(3): 512-521.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2019.03.012        http://www.zjujournals.com/eng/CN/Y2019/V53/I3/512

图 1  融合图像特征的深度感知处理和神经网络预测(DPPNNP)方法原理图
失真类型参数设置 指标 MSSIM IFC VIF GSM FSIM VSI SVD IGM MAD DPPNNP
JPEG2000
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{4}}}&{{\gamma _{{\rm{12}}}} = 0.{0}1} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.02}}}&{{\gamma _{{\rm{22}}}} = 0.0{01}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{00}}1} \end{array}$
RMSE 4.414 2 4.696 1 4.714 8 4.179 3 4.408 6 5.496 8 4.445 8 3.310 5 3.059 1 3.873 1
PLCC 0.983 3 0.981 1 0.980 9 0.985 1 0.983 4 0.974 0 0.983 1 0.990 7 0.992 0 0.987 2
SROCC 0.984 5 0.964 1 0.978 7 0.986 9 0.988 4 0.969 8 0.981 4 0.986 6 0.987 1 0.984 1
JPEG
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{3}}}&{{\gamma _{{\rm{12}}}} = 0.1} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.02}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.001}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{10}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{4}}} \end{array}$
RMSE 3.600 4 3.251 14 2.802 7 3.909 6 3.861 5 3.941 4 3.419 4 2.336 6 4.532 8 1.936 9
PLCC 0.989 3 0.991 3 0.993 6 0.987 4 0.987 7 0.987 2 0.990 4 0.995 5 0.983 0 0.996 9
SROCC 0.977 9 0.953 8 0.965 0 0.965 0 0.980 1 0.954 1 0.953 6 0.976 0 0.967 1 0.975 8
WN
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = {0}{\rm{.4}}}&{{\gamma _{{\rm{12}}}} = 0.{01}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.04}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.01}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = {0}{\rm{.000\;1}}} \end{array}$
RMSE 2.866 7 3.078 6 4.131 3 3.307 9 3.933 6 3.335 2 2.374 2 1.659 0 3.124 3 1.260 6
PLCC 0.992 8 0.991 7 0.985 1 0.990 5 0.986 5 0.990 3 0.995 1 0.997 6 0.991 5 0.998 6
SROCC 0.984 7 0.957 7 0.966 6 0.987 9 0.976 9 0.974 7 0.994 0 0.994 5 0.991 3 0.993 3
Gblur
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = {0}{\rm{.4}}}&{{\gamma _{{\rm{12}}}} = 0.{01}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.04}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.01}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = {0}{\rm{.000\;1}}} \end{array}$
RMSE 5.454 9 3.859 5 7.503 8 5.505 4 4.566 3 5.523 8 6.776 2 3.629 6 3.893 0 3.135 4
PLCC 0.976 3 0.988 2 0.954 6 0.975 8 0.983 4 0.975 7 0.963 1 0.989 6 0.988 0 0.992 2
SROCC 0.989 2 0.969 0 0.933 4 0.987 9 0.991 6 0.967 9 0.963 5 0.988 8 0.987 7 0.987 7
Fastfading
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{3}}}&{{\gamma _{{\rm{12}}}} = 0.{\rm{1}}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.02}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.001}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{40}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{00}}1} \end{array}$
RMSE 4.818 3 5.117 4 4.125 8 5.078 8 5.258 1 5.553 9 6.184 3 5.086 5 4.192 0 2.838 3
PLCC 0.979 7 0.977 1 0.985 2 0.977 5 0.975 8 0.973 0 0.966 4 0.977 4 0.984 7 0.993 6
SROCC 0.984 1 0.957 0 0.969 5 0.982 7 0.981 5 0.964 4 0.969 9 0.972 0 0.985 0 0.990 7
失真类型总和
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = {0}{\rm{.28}}}&{{\gamma _{{\rm{12}}}} = 0.{\rm{3}}} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.4}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.001}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{6}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = 0.{\rm{08}}} \end{array}$
RMSE 6.231 9 5.371 8 7.469 2 6.103 1 5.784 3 6.597 9 6.279 0 4.722 0 5.517 2 4.132 4
PLCC 0.967 3 0.975 8 0.952 6 0.968 6 0.971 9 0.963 2 0.966 8 00.981 3 0.974 4 0.987 5
SROCC 0.965 1 0.955 2 0.920 6 0.971 8 0.977 8 0.949 1 0.967 2 0.974 1 0.977 3 0.981 5
表 2  提出的DPPNNP与其他方法在LIVE数据库上的评价结果比较
图 2  BP神经网络多通道评价融合预测模型
失真类型 指标 E F Q
BP模型 CSF函数 BP模型 CSF函数 BP模型 CSF函数
JPEG2000 RMSE 5.260 7 11.617 2 4.554 0 5.308 6 4.063 1 4.362 6
PLCC 0.976 2 0.878 0 0.982 2 0.975 8 0.985 9 0.983 7
SROCC 0.960 9 0.611 0 0.972 7 0.967 9 0.983 3 0.986 3
JPEG RMSE 3.304 9 9.356 4 2.893 8 3.495 9 3.567 8 3.544 2
PLCC 0.991 0 0.926 5 0.993 1 0.990 0 0.986 7 0.989 7
SROCC 0.952 5 0.736 9 0.964 5 0.931 0 0.977 2 0.976 0
WN RMSE 3.366 2 10.333 7 3.349 8 3.295 3 1.331 4 2.427 7
PLCC 0.990 1 0.902 6 0.990 2 0.990 5 0.998 5 0.994 9
SROCC 0.975 1 0.683 9 0.986 0 0.983 5 0.992 8 0.993 2
gblur RMSE 6.173 2 6.982 4 5.527 6 5.513 4 3.013 2 5.196 6
PLCC 0.969 5 0.960 8 0.975 6 0.975 8 0.992 8 0.978 5
SROCC 0.953 1 0.958 3 0.983 4 0.975 8 0.988 0 0.987 7
Fastfading RMSE 6.916 8 11.155 8 6.241 0 7.739 9 3.731 1 6.591 0
PLCC 0.957 8 0.886 0 0.965 8 0.946 8 0.987 9 0.961 7
SROCC 0.931 8 0.741 2 0.960 7 0.923 2 0.986 3 0.956 6
表 1  BP神经网络预测结果和CSF融合结果比较
数据库名称及参数设置 指标 MSSIM IFC VIF GSM FSIM VSI SVD IGM MAD DPPNNP
CSIQ
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{001}}}&{{\gamma _{{\rm{12}}}} = 0.1} \\ {{\gamma _{{\rm{21}}}} = {0}{\rm{.3}}}&{{\gamma _{{\rm{22}}}} = {\rm{4}}{\rm{.0}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{100}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = {\rm{1}}{\rm{.8}}} \end{array}$
RMSE 0.118 0 0.179 9 0.142 4 0.172 1 0.110 1 0.099 0 0.170 7 0.099 6 0.084 1 0.071 8
PLCC 0.879 0 0.738 9 0.845 9 0.764 6 0.911 1 0.928 7 0.768 9 0.928 7 0.949 2 0.963 2
SROCC 0.913 0 0.767 6 0.922 1 0.908 9 0.928 9 0.945 3 0.753 0 0.941 1 0.937 2 0.958 8
tid2013
$\begin{array}{*{20}{c}} {{\gamma _{{\rm{11}}}} = 0.{\rm{28}}}&{{\gamma _{{\rm{12}}}} = 0.{\rm{8}}} \\ {{\gamma _{{\rm{21}}}} = {\rm{1}}{\rm{.4}}}&{{\gamma _{{\rm{22}}}} = {0}{\rm{.000\;1}}} \\ {{\gamma _{{\rm{31}}}} = {\rm{4}}{\rm{.0}}}&{{\gamma _{{\rm{32}}}} = {\rm{100}}{\rm{.0}}} \end{array}$
RMSE 0.766 9 1.217 1 0.974 9 0.971 7 0.710 8 0.678 1 0.933 0 0.692 8 0.704 2 0.636 0
PLCC 0.782 3 0.150 5 0.610 7 0.802 4 0.816 5 0.834 6 0.652 5 0.826 6 0.820 3 0.856 2
SROCC 0.788 8 0.513 9 0.609 5 0.796 6 0.796 8 0.889 0 0.632 7 0.802 6 0.750 7 0.802 9
表 3  所提DPPNNP与其他方法在CSIQ和tid2013数据库的评价结果比较
图 3  不同方法散点分布图
评价方法 t/s 评价方法 t/s
MSSIM 171 IFC 2 020
SVD 238 VIF 2 083
VSI 484 IGM 39 194
FSIM 653 DPPNNP 21 127
MAD 3 128
表 4  不同图像质量评价方法的时间复杂度测试结果比较
1 GUO Y C, HAO Y T, YU M Image retargeting quality assessment based on content deformation measurement[J]. Signal Processing: Image Communication, 2018, 67 (6): 171- 181
2 CHEN Z B, LIN J X, LIAO N, et al Full reference quality assessment for image retargeting based on natural scene statistics modeling and bi-directional saliency similarity[J]. IEEE Transactions on Image Processing, 2017, 26 (11): 5138- 5148
doi: 10.1109/TIP.2017.2736422
3 ZHANG Y CH, KING N N, MA L, et al Objective quality assessment of image retargeting by incorporating fidelity measures and inconsistency detection[J]. IEEE Transactions on Image Processing, 2017, 26 (11): 5980- 5993
4 ZHANG Y B, LIN W S, LI Q H, et al Multiple-level feature-based measure for retargeted image quality[J]. IEEE Transactions on Image Processing, 2018, 27 (1): 451- 463
doi: 10.1109/TIP.2017.2761556
5 丰明坤, 王中鹏, 叶绿 视觉稀疏化多通道多特征自适应的图像评价[J]. 仪器仪表学报, 2016, 37 (3): 667- 674
FENG Ming-kun, WANG ZHong-peng, YE Lv Image quality assessment based on adaptive sparse visual multi-channel and multi-feature pooling[J]. Chinese Journal of Scientific Instrument, 2016, 37 (3): 667- 674
doi: 10.3969/j.issn.0254-3087.2016.03.025
6 BAMPIS C G, GUPTA P, SOUNDARARAJAN R, et al SpEED-QA: patial efficient entropic differencing for image and video quality[J]. IEEE Signal Processing Letters, 2017, 24 (9): 1333- 1337
doi: 10.1109/LSP.2017.2726542
7 NI ZH K, MA L, ZENG H Q, et al Gradient direction for screen content image quality assessment[J]. IEEE Signal Processing Letters, 2016, 23 (10): 1394- 1398
doi: 10.1109/LSP.2016.2599294
8 KIM J, ZENG H, GHADIYARAM D, et al Deep convolutional neural models for picture-quality prediction: challenges and solutions to data-driven image quality assessment[J]. IEEE Signal Processing Magazine, 2017, 34 (6): 130- 141
doi: 10.1109/MSP.2017.2736018
9 GHADIYARAM D, BOVIK A. C Massive online crowdsourced study of subjective and objective picture quality[J]. IEEE Transactions on Image Process, 2016, 25 (1): 372- 387
doi: 10.1109/TIP.2015.2500021
10 LARSON E C, CHANDLER D M Most apparent distortion: full-reference image quality assessment and the role of strategy[J]. Journal of Electronic Imaging, 2010, 19 (1): 011006-1- 011006-21
11 ZHANG L, ZHANG L, MOU X Q, et al FSIM: a feature similarity index for image quality assessment[J]. IEEE Transactions on Image Processing, 2011, 20 (8): 2378- 2386
doi: 10.1109/TIP.2011.2109730
12 ZHANG L, SHEN Y LI H Y VSI: a visual saliency-induced index for perceptual image quality assessment[J]. IEEE Transactions on Image Processing, 2014, 23 (10): 4270- 4281
doi: 10.1109/TIP.2014.2346028
13 DING Y, WANG SH Z, ZHANG D Full-reference image quality assessment using statistical local correlation[J]. Electronics Letters, 2014, 50 (2): 79- 81
doi: 10.1049/el.2013.3365
14 WU J J, LIN W S, SHI G M Perceptual quality metric with internal generative mechanism[J]. IEEE Transactions on Image Processing, 2013, 22 (1): 43- 54
doi: 10.1109/TIP.2012.2214048
15 丰明坤, 赵生妹, 邢超 基于视觉显著失真度的图像质量自适应评价方法[J]. 电子与信息学报, 2015, 37 (9): 2062- 2068
FENG Ming-kun, ZHAO SHeng-mei, XING CHao Image quality self-adaptive assessment based on visual salience distortion[J]. Journal of Electronics & Information Technology, 2015, 37 (9): 2062- 2068
16 丰明坤, 赵生妹, 施祥 视觉多通道梯度与低阶矩自适应图像评价[J]. 仪器仪表学报, 2015, 36 (11): 2531- 2537
FENG Ming-kun, ZHAO SHeng-mei, SHI Xiang Adaptive image quality assessment based on visual multi-channel gradient and low order moment[J]. Chinese Journal of Scientific Instrument, 2015, 36 (11): 2531- 2537
doi: 10.3969/j.issn.0254-3087.2015.11.017
17 WANG ZH, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612
doi: 10.1109/TIP.2003.819861
18 WANG ZH, LU L G, BOVIK A C Foveation scalable video coding with automatic fixation selection[J]. IEEE Transactions on Image Processing, 2003, 12 (2): 243- 254
doi: 10.1109/TIP.2003.809015
19 LIU A M, LIN W S, NARWARIA M Image quality assessment based on gradient similarity[J]. IEEE Transaction on Image Processing, 2012, 21 (4): 1500- 1512
doi: 10.1109/TIP.2011.2175935
20 GAO X B, LU W, TAO D CH, et al Image quality assessment based on multiscale geometric analysis[J]. IEEE Transactions on Image Processing, 2009, 18 (7): 1409- 1423
doi: 10.1109/TIP.2009.2018014
21 林志洁, 丰明坤 深度视觉特征与策略互补融合的图像质量评价[J]. 模式识别与人工智能, 2017, 30 (8): 682- 691
LIN ZHi-jie, FENG Ming-kun Image quality assessment based on complementary pooling of deeply visual feature and strategy[J]. Pattern Recognition and Artificial Intelligence, 2017, 30 (8): 682- 691
22 SHNAYDERMAN A, GUSEV A, ESKICIOGLU A M An SVD-based grayscale image quality measure for local and global assessment[J]. IEEE Transactions on Image Processing, 2006, 15 (2): 422- 429
doi: 10.1109/TIP.2005.860605
23 HU A ZH, ZHANG R YIN D, et al Image quality assessment using an SVD-based structural projection[J]. Signal Processing: Image Communication, 2014, 29 (3): 293- 302
doi: 10.1016/j.image.2014.01.007
24 WANG ZH, SIMONCELLI E P, BOVIK A C. Multi-scale structural similarity for image quality assessment [C]// Proceedings of the 37th IEEE Asilomar Conference on Signals, Systems and Computers. Pacific Grove: IEEE, 2002(2): 1398–1402.
25 SHEIKH H R, BOVIK A C, VECIANA G D An information fidelity criterion for image quality assessment using natural scene statistics[J]. IEEE Transactions on Image Processing, 2005, 14 (12): 2117- 2128
doi: 10.1109/TIP.2005.859389
[1] 粟序明,方成刚,潘裕斌,吴伟伟,李亚萍,朱浪. 变照度下的视觉测量系统误差建模[J]. 浙江大学学报(工学版), 2020, 54(10): 1929-1935.
[2] 谢杰, 黄筱调, 方成刚, 周宝仓, 陆宁. 磨齿机电主轴热特性及热误差建模[J]. 浙江大学学报(工学版), 2018, 52(2): 247-254.
[3] 孟濬, 邓晓雨, 虞捷舟. 基于变量聚类的BP神经网络术后生存期预测模型[J]. 浙江大学学报(工学版), 2018, 52(12): 2365-2371.
[4] 尤海辉, 马增益, 唐义军, 王月兰, 郑林, 俞钟, 吉澄军. 循环流化床入炉垃圾热值软测量[J]. 浙江大学学报(工学版), 2017, 51(6): 1163-1172.
[5] 巫江虹, 薛志强, 金鹏, 李会喜. 电动汽车热泵空调微通道换热器温度分布特性[J]. 浙江大学学报(工学版), 2016, 50(8): 1537-1544.
[6] 崔光茫,冯华君,徐之海,李奇,陈跃庭. 基于CSF和仿射重建模型的噪声图像质量评价[J]. 浙江大学学报(工学版), 2016, 50(1): 144-150.
[7] 刘磊,杨鹏,刘作军. 基于多源信息和粒子群优化算法的下肢运动模式识别[J]. 浙江大学学报(工学版), 2015, 49(3): 439-447.
[8] 黄晓烁,何衍,蒋静坪. 基于互联网无刷直流电机传动系统的控制策略[J]. J4, 2013, 47(5): 831-836.
[9] 王翔,丁勇. 基于Gabor滤波器的全参考图像质量评价方法[J]. J4, 2013, 47(3): 422-430.
[10] 童水光, 王相兵, 钟崴, 张健. 基于BP-HGA的起重机刚性支腿动态优化设计[J]. J4, 2013, 47(1): 122-130.
[11] 王德明, 王莉, 张广明. 基于遗传BP神经网络的短期风速预测模型[J]. J4, 2012, 46(5): 837-841.
[12] 赵振, 张树有. 全分断周期混合电流模型及其应用[J]. J4, 2012, 46(2): 301-308.
[13] 楼斌, 沈海斌, 赵武锋, 等. 基于自然图像统计的无参考图像质量评价[J]. J4, 2010, 44(2): 248-252.
[14] 楼斌, 沈海斌, 赵武锋, 等. 基于失真模型的结构相似度图像质量评价[J]. J4, 2009, 43(5): 864-868.
[15] 徐敬华, 张树有. 基于形态分布图与BP神经网络的三维模型检索方法[J]. J4, 2009, 43(5): 877-883.