Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2019, Vol. 53 Issue (4): 770-776    DOI: 10.3785/j.issn.1008-973X.2019.04.018
自动化技术     
基于融合约束局部模型的三维人脸特征点定位
成翔昊1,2(),达飞鹏1,2,*(),汪亮1,2
1. 东南大学 自动化学院，江苏 南京 210096
2. 东南大学 复杂工程系统测量与控制教育部重点实验室，江苏 南京 210096
Feature fusion based constrained local model for three-dimensional facial landmark localization
Xiang-hao CHENG1,2(),Fei-peng DA1,2,*(),Liang WANG1,2
1. School of Automation, Southeast University, Nanjing 210096, China
2. Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Southeast University, Nanjing 210096, China
 全文: PDF(722 KB)   HTML
摘要：

提出基于特征融合约束局部模型的三维人脸特征点定位算法. 该算法对每个特征点分别使用三维网格的深度信息和网格局部形状信息训练分类器，对分类器的响应进行融合. 使用基于融合响应的正则化特征点均值漂移算法进行模型拟合，实现特征点定位. 三维人脸特征点定位经常需要对每个特征点的候选点集进行遍历产生候选点组合，该算法使用模型拟合代替穷举搜索，避免了嵌套循环带来的快速增长的时间开销. 使用FRGC v2.0和Bosphorus数据库，对算法进行实验评估. FRGC v2.0库上的特征点平均误差为2.48~4.12 mm，总体检测成功率为97.3%，其中中性、温和及极端表情下的检测成功率分别为97.6%、97.4%和95.5%. Bosphorus库上3种姿态下的检测成功率分别是94%、95%和89%. 实验结果表明，提出方法具有较好的效果，对表情和小幅度的姿态变化具有较好的鲁棒性.
关键词： 三维人脸特征点定位;  约束局部模型;  特征融合;  深度信息;  局部形状信息;  正则化特征点均值漂移    
Abstract:

An algorithm for automatic detection of landmarks on three-dimensional faces was proposed by using a feature fusion based constrained local model. A classifier based on depth information and a classifier based on local shape information of three-dimensional meshes were trained for each landmark. The responses of two classifiers were merged and the regularized landmark mean shift algorithm was applied on fitting for the localization of landmarks. Traversing candidates of each landmark is usually necessary in three-dimensional facial landmark localization to generate candidate combinations. The problem of time overhead for nested loops that increases rapidly by using model fitting instead of exhaustive search was solved. The approach was evaluated based on three-dimensional face databases: FRGC v2.0 and Bosphorus. The mean error of every landmark in the FRGC v2.0 is between 2.48 mm to 4.12 mm. The overall detection success rate is 97.3%, among which 97.6% for neutral expression, 97.4% for mild, 95.5% for extreme. On the Bosphorus database, the success rate of 94%, 95% and 89% was respectively achieved under three different poses. The experimental results show that the proposed approach is comparable to state-of-the-art methods in terms of its accuracy, and good robustness is achieved against expression and small pose variation.
Key words: three-dimensional facial landmark localization    constrained local model    feature fusion    depth information    local shape information    regularized landmark mean shift
收稿日期: 2018-03-15 出版日期: 2019-03-28
CLC:  TP 391  
通讯作者: 达飞鹏     E-mail: 220151407@seu.edu.cn;dafp@seu.edu.cn
作者简介: 成翔昊(1993—)，男，硕士，从事三维人脸特征点定位与识别研究. orcid.org/0000-0003-1100-742X. E-mail： 220151407@seu.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
成翔昊
达飞鹏
汪亮

引用本文:

成翔昊,达飞鹏,汪亮. 基于融合约束局部模型的三维人脸特征点定位[J]. 浙江大学学报(工学版), 2019, 53(4): 770-776.

Xiang-hao CHENG,Fei-peng DA,Liang WANG. Feature fusion based constrained local model for three-dimensional facial landmark localization. Journal of ZheJiang University (Engineering Science), 2019, 53(4): 770-776.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2019.04.018    或    http://www.zjujournals.com/eng/CN/Y2019/V53/I4/770

图 1  基于特征融合约束局部模型的三维人脸特征点算法的总体框架
图 2  三维人脸14个特征点的位置
检测误差/mm 文献[14]方法 文献[15]方法 文献[21]方法 本文方法
平均误差 标准差 平均误差 标准差 平均误差 标准差 平均误差 标准差 成功率/%
0 5.87 3.11 4.49 2.64 3.04 2.00 2.85 1.48 96.1
1 4.31 2.44 3.35 1.63 2.10 1.46 2.48 1.55 97.0
2 4.20 2.07 2.55 1.60 — — 2.90 1.83 94.3
3 4.29 2.03 3.35 1.63 2.28 1.55 2.53 1.61 96.5
4 6.00 3.03 4.49 2.64 4.13 2.36 2.77 2.04 93.7
5 3.35 2.00 2.22 1.31 3.34 2.41 3.58 1.97 87.4
6 4.73 3.68 3.09 1.18 7.77 4.03 3.14 2.22 91.1
7 4.86 3.54 3.09 1,18 7.61 3.96 3.37 2.39 89.6
8 3.67 3.11 2.81 1.11 — — 3.16 2.56 90.5
9 5.47 3.45 4.05 3.12 4.50 3.85 2.96 1.69 95.7
10 5.64 3.58 4.05 3.12 4.37 3.82 2.73 1.84 94.7
11 4.23 3.21 3.40 1.97 3.66 3.52 3.25 2.33 90.0
12 5.46 3.92 4.82 4.04 5.49 5.59 3.87 2.98 80.2
13 7.28 7.41 5.39 4.01 6.45 5.60 4.12 3.26 76.3
总体 4.95 3.33 3.44 — 4.97 — 3.12 1.37 97.3
表 1  基于FRGCv2.0数据库的实验结果以及与其他算法的精度对比
1 郭梦丽, 达飞鹏, 邓星, 等 基于关键点和局部特征的三维人脸识别[J]. 浙江大学学报: 工学版, 2017, 51 (3): 584- 589
GUO Meng-li, DA Fei-peng, DENG Xing, et al 3D face recognition based on keypoints and local feature[J]. Journal of Zhejiang University: Engineering Science, 2017, 51 (3): 584- 589
2 汤兰兰, 盖绍彦, 达飞鹏, 等 基于网格纵横局部二值模式的三维人脸识别[J]. 仪器仪表学报, 2016, 37 (6): 1413- 1420
TANG Lan-lan, GAI Shao-yan, DA Fei-peng, et al A 3D face recognition method based on the local binary pattern from vertical and horizontal on the mesh[J]. Chinese Journal of Scientific Instrument, 2016, 37 (6): 1413- 1420
doi: 10.3969/j.issn.0254-3087.2016.06.027
3 DENG X, DA F, SHAO H Efficient 3D face recognition using local covariance descriptor and Riemannian kernel sparse coding[J]. Computers and Electrical Engineering, 2017, 62 (8): 81- 91
4 DERKACH D, SUKNO F M. Local shape spectrum analysis for 3D facial expression recognition [C] // Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG). Washington DC: IEEE, 2017: 41–47.
5 ZHAO Q, OKADA K, ROSENBAUM K, et al Digital facial dysmorphology for genetic screening: Hierarchical constrained local model using ICA[J]. Medical Image Analysis, 2014, 18 (5): 699- 710
doi: 10.1016/j.media.2014.04.002
6 COOTES T F, TAYLOR C J, COOPER D H, et al Active shape models-their training and application[J]. Computer Vision and Image Understanding, 1995, 61 (1): 38- 59
doi: 10.1006/cviu.1995.1004
7 EDWARDS G J, COOTES T F, TAYLOR C J. Face recognition using active appearance models [C] // Proceedings of European Conference on Computer Vision (ECCV). Berlin: Springer, 1998: 581–595.
8 CRISTINACCE D, COOTES T Automatic feature localisation with constrained local models[J]. Pattern Recognition, 2008, 41 (10): 3054- 3067
doi: 10.1016/j.patcog.2008.01.024
9 BALTRU?AITIS T, ROBINSON P, MORENCY L P. 3D constrained local model for rigid and non-rigid facial tracking [C] // Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Rhode Island: IEEE, 2012: 2610–2617.
10 SARAGIH J M, LUCEY S, COHN J F Deformable model fitting by regularized landmark mean-shift[J]. International Journal of Computer Vision, 2011, 91 (2): 200- 215
doi: 10.1007/s11263-010-0380-4
11 CHENG S, ZAFEIRIOU S, ASTHANA A, et al. 3D facial geometric features for constrained local model [C] // Proceedings of IEEE International Conference on Image Processing (ICIP). Paris: IEEE, 2014: 1425–1429.
12 NAIR P, CAVALLARO A 3-D face detection, landmark localization, and registration using a point distribution model[J]. IEEE Transactions on Multimedia, 2009, 11 (4): 611- 623
doi: 10.1109/TMM.2009.2017629
13 PERAKIS P, PASSALIS G, THEOHARIS T, et al 3D facial landmark detection under large yaw and expression variations[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (7): 1552- 1564
doi: 10.1109/TPAMI.2012.247
14 CREUSOT C, PEARS N, AUSTIN J A machine-learning approach to keypoint detection and landmarking on 3D meshes[J]. International Journal of Computer Vision, 2013, 102 (1-3): 146- 179
doi: 10.1007/s11263-012-0605-9
15 SUKNO F M, WADDINGTON J L, WHELAN P F 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features[J]. IEEE Transactions on Cybernetics, 2015, 45 (9): 1717- 1730
doi: 10.1109/TCYB.2014.2359056
16 GOWER J C. Generalized procrustes analysis [J]. Psychometrika, 1975, 40(1): 33-51.
17 MOGHADDAM B, PENTLAND A Probabilistic visual learning for object representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19 (7): 696- 710
doi: 10.1109/34.598227
18 JOHNSON A E. Spin-images: a representation for 3D surface matching [D]. Pittsburgh: Carnegie Mellon University, 1997.
19 PHILLIPS P J, FLYNN P J, SCRUGGS T, et al. Overview of the face recognition grand challenge [C] // Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). San Diego: IEEE, 2005: 947–954.
20 SAVRAN A, ALYüZ N, DIBEKLIO?LU H, et al. Bosphorus database for 3D face analysis [C] // Proceedings of European Workshop on Biometrics and Identity Management (BIOID). Berlin: Springer, 2008: 47–56.
[1] 郑浦,白宏阳,李伟,郭宏伟. 复杂背景下的小目标检测算法[J]. 浙江大学学报(工学版), 2020, 54(9): 1777-1784.
[2] 周登文,田金月,马路遥,孙秀秀. 基于多级特征并联的轻量级图像语义分割[J]. 浙江大学学报(工学版), 2020, 54(8): 1516-1524.
[3] 林泓,卢瑶瑶. 聚焦难样本的区分尺度的文字检测方法[J]. 浙江大学学报(工学版), 2019, 53(8): 1506-1516.
[4] 王万良,杨胜兰,赵燕伟,李卓蓉. 基于条件边界平衡生成对抗网络的河流表面流速估测[J]. 浙江大学学报(工学版), 2019, 53(11): 2118-2128.
[5] 赖小波,朱世强,方纯洁. 一种复杂背景图像三维重建算法及其医学应用[J]. J4, 2012, 46(11): 2061-2067.
[6] 鲍必赛, 伍健荣, 楼晓俊, 刘海涛. 基于二维特征矩阵的特征融合算法[J]. J4, 2012, 46(11): 2081-2088.
[7] 姚拓中 项志宇 刘济林. 基于图像多特征融合的野外水体障碍物检测[J]. , 2009, 43(4): 605-609.
[8] 刘康苗 仇光 卜佳俊 陈纯 周纯. 基于视觉和语义融合特征的阶段式图像聚类[J]. J4, 2008, 42(12): 2043-2048.