Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2019, Vol. 53 Issue (4): 770-776    DOI: 10.3785/j.issn.1008-973X.2019.04.018
    
Feature fusion based constrained local model for three-dimensional facial landmark localization
Xiang-hao CHENG1,2(),Fei-peng DA1,2,*(),Liang WANG1,2
1. School of Automation, Southeast University, Nanjing 210096, China
2. Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education, Southeast University, Nanjing 210096, China
Download: HTML     PDF(722KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

An algorithm for automatic detection of landmarks on three-dimensional faces was proposed by using a feature fusion based constrained local model. A classifier based on depth information and a classifier based on local shape information of three-dimensional meshes were trained for each landmark. The responses of two classifiers were merged and the regularized landmark mean shift algorithm was applied on fitting for the localization of landmarks. Traversing candidates of each landmark is usually necessary in three-dimensional facial landmark localization to generate candidate combinations. The problem of time overhead for nested loops that increases rapidly by using model fitting instead of exhaustive search was solved. The approach was evaluated based on three-dimensional face databases: FRGC v2.0 and Bosphorus. The mean error of every landmark in the FRGC v2.0 is between 2.48 mm to 4.12 mm. The overall detection success rate is 97.3%, among which 97.6% for neutral expression, 97.4% for mild, 95.5% for extreme. On the Bosphorus database, the success rate of 94%, 95% and 89% was respectively achieved under three different poses. The experimental results show that the proposed approach is comparable to state-of-the-art methods in terms of its accuracy, and good robustness is achieved against expression and small pose variation.



Key wordsthree-dimensional facial landmark localization      constrained local model      feature fusion      depth information      local shape information      regularized landmark mean shift     
Received: 15 March 2018      Published: 28 March 2019
CLC:  TP 391  
Corresponding Authors: Fei-peng DA     E-mail: 220151407@seu.edu.cn;dafp@seu.edu.cn
Cite this article:

Xiang-hao CHENG,Fei-peng DA,Liang WANG. Feature fusion based constrained local model for three-dimensional facial landmark localization. Journal of ZheJiang University (Engineering Science), 2019, 53(4): 770-776.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2019.04.018     OR     http://www.zjujournals.com/eng/Y2019/V53/I4/770


基于融合约束局部模型的三维人脸特征点定位

提出基于特征融合约束局部模型的三维人脸特征点定位算法. 该算法对每个特征点分别使用三维网格的深度信息和网格局部形状信息训练分类器,对分类器的响应进行融合. 使用基于融合响应的正则化特征点均值漂移算法进行模型拟合,实现特征点定位. 三维人脸特征点定位经常需要对每个特征点的候选点集进行遍历产生候选点组合,该算法使用模型拟合代替穷举搜索,避免了嵌套循环带来的快速增长的时间开销. 使用FRGC v2.0和Bosphorus数据库,对算法进行实验评估. FRGC v2.0库上的特征点平均误差为2.48~4.12 mm,总体检测成功率为97.3%,其中中性、温和及极端表情下的检测成功率分别为97.6%、97.4%和95.5%. Bosphorus库上3种姿态下的检测成功率分别是94%、95%和89%. 实验结果表明,提出方法具有较好的效果,对表情和小幅度的姿态变化具有较好的鲁棒性.


关键词: 三维人脸特征点定位,  约束局部模型,  特征融合,  深度信息,  局部形状信息,  正则化特征点均值漂移 
Fig.1 Framework of three-dimensional facial landmark localization algorithm based on feature fusion based constrained local model
Fig.2 Positions of 14 three-dimensional facial landmarks
检测误差/mm 文献[14]方法 文献[15]方法 文献[21]方法 本文方法
平均误差 标准差 平均误差 标准差 平均误差 标准差 平均误差 标准差 成功率/%
0 5.87 3.11 4.49 2.64 3.04 2.00 2.85 1.48 96.1
1 4.31 2.44 3.35 1.63 2.10 1.46 2.48 1.55 97.0
2 4.20 2.07 2.55 1.60 2.90 1.83 94.3
3 4.29 2.03 3.35 1.63 2.28 1.55 2.53 1.61 96.5
4 6.00 3.03 4.49 2.64 4.13 2.36 2.77 2.04 93.7
5 3.35 2.00 2.22 1.31 3.34 2.41 3.58 1.97 87.4
6 4.73 3.68 3.09 1.18 7.77 4.03 3.14 2.22 91.1
7 4.86 3.54 3.09 1,18 7.61 3.96 3.37 2.39 89.6
8 3.67 3.11 2.81 1.11 3.16 2.56 90.5
9 5.47 3.45 4.05 3.12 4.50 3.85 2.96 1.69 95.7
10 5.64 3.58 4.05 3.12 4.37 3.82 2.73 1.84 94.7
11 4.23 3.21 3.40 1.97 3.66 3.52 3.25 2.33 90.0
12 5.46 3.92 4.82 4.04 5.49 5.59 3.87 2.98 80.2
13 7.28 7.41 5.39 4.01 6.45 5.60 4.12 3.26 76.3
总体 4.95 3.33 3.44 4.97 3.12 1.37 97.3
Tab.1 Experimental results based on FRGCv2.0 database and accuracy comparison with other algorithms
[1]   郭梦丽, 达飞鹏, 邓星, 等 基于关键点和局部特征的三维人脸识别[J]. 浙江大学学报: 工学版, 2017, 51 (3): 584- 589
GUO Meng-li, DA Fei-peng, DENG Xing, et al 3D face recognition based on keypoints and local feature[J]. Journal of Zhejiang University: Engineering Science, 2017, 51 (3): 584- 589
[2]   汤兰兰, 盖绍彦, 达飞鹏, 等 基于网格纵横局部二值模式的三维人脸识别[J]. 仪器仪表学报, 2016, 37 (6): 1413- 1420
TANG Lan-lan, GAI Shao-yan, DA Fei-peng, et al A 3D face recognition method based on the local binary pattern from vertical and horizontal on the mesh[J]. Chinese Journal of Scientific Instrument, 2016, 37 (6): 1413- 1420
doi: 10.3969/j.issn.0254-3087.2016.06.027
[3]   DENG X, DA F, SHAO H Efficient 3D face recognition using local covariance descriptor and Riemannian kernel sparse coding[J]. Computers and Electrical Engineering, 2017, 62 (8): 81- 91
[4]   DERKACH D, SUKNO F M. Local shape spectrum analysis for 3D facial expression recognition [C] // Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG). Washington DC: IEEE, 2017: 41–47.
[5]   ZHAO Q, OKADA K, ROSENBAUM K, et al Digital facial dysmorphology for genetic screening: Hierarchical constrained local model using ICA[J]. Medical Image Analysis, 2014, 18 (5): 699- 710
doi: 10.1016/j.media.2014.04.002
[6]   COOTES T F, TAYLOR C J, COOPER D H, et al Active shape models-their training and application[J]. Computer Vision and Image Understanding, 1995, 61 (1): 38- 59
doi: 10.1006/cviu.1995.1004
[7]   EDWARDS G J, COOTES T F, TAYLOR C J. Face recognition using active appearance models [C] // Proceedings of European Conference on Computer Vision (ECCV). Berlin: Springer, 1998: 581–595.
[8]   CRISTINACCE D, COOTES T Automatic feature localisation with constrained local models[J]. Pattern Recognition, 2008, 41 (10): 3054- 3067
doi: 10.1016/j.patcog.2008.01.024
[9]   BALTRU?AITIS T, ROBINSON P, MORENCY L P. 3D constrained local model for rigid and non-rigid facial tracking [C] // Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Rhode Island: IEEE, 2012: 2610–2617.
[10]   SARAGIH J M, LUCEY S, COHN J F Deformable model fitting by regularized landmark mean-shift[J]. International Journal of Computer Vision, 2011, 91 (2): 200- 215
doi: 10.1007/s11263-010-0380-4
[11]   CHENG S, ZAFEIRIOU S, ASTHANA A, et al. 3D facial geometric features for constrained local model [C] // Proceedings of IEEE International Conference on Image Processing (ICIP). Paris: IEEE, 2014: 1425–1429.
[12]   NAIR P, CAVALLARO A 3-D face detection, landmark localization, and registration using a point distribution model[J]. IEEE Transactions on Multimedia, 2009, 11 (4): 611- 623
doi: 10.1109/TMM.2009.2017629
[13]   PERAKIS P, PASSALIS G, THEOHARIS T, et al 3D facial landmark detection under large yaw and expression variations[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (7): 1552- 1564
doi: 10.1109/TPAMI.2012.247
[14]   CREUSOT C, PEARS N, AUSTIN J A machine-learning approach to keypoint detection and landmarking on 3D meshes[J]. International Journal of Computer Vision, 2013, 102 (1-3): 146- 179
doi: 10.1007/s11263-012-0605-9
[15]   SUKNO F M, WADDINGTON J L, WHELAN P F 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features[J]. IEEE Transactions on Cybernetics, 2015, 45 (9): 1717- 1730
doi: 10.1109/TCYB.2014.2359056
[16]   GOWER J C. Generalized procrustes analysis [J]. Psychometrika, 1975, 40(1): 33-51.
[17]   MOGHADDAM B, PENTLAND A Probabilistic visual learning for object representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19 (7): 696- 710
doi: 10.1109/34.598227
[18]   JOHNSON A E. Spin-images: a representation for 3D surface matching [D]. Pittsburgh: Carnegie Mellon University, 1997.
[19]   PHILLIPS P J, FLYNN P J, SCRUGGS T, et al. Overview of the face recognition grand challenge [C] // Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). San Diego: IEEE, 2005: 947–954.
[20]   SAVRAN A, ALYüZ N, DIBEKLIO?LU H, et al. Bosphorus database for 3D face analysis [C] // Proceedings of European Workshop on Biometrics and Identity Management (BIOID). Berlin: Springer, 2008: 47–56.
[1] Pu ZHENG,Hong-yang BAI,Wei LI,Hong-wei GUO. Small target detection algorithm in complex background[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1777-1784.
[2] Deng-wen ZHOU,Jin-yue TIAN,Lu-yao MA,Xiu-xiu SUN. Lightweight image semantic segmentation based on multi-level feature cascaded network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(8): 1516-1524.
[3] Hong LIN,Yao-yao LU. Scale differentiated text detection method focusing on hard examples[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(8): 1506-1516.
[4] Wan-liang WANG,Sheng-lan YANG,Yan-wei ZHAO,Zhuo-rong LI. Estimation of river surface flow velocity based on conditional boundary equilibrium generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(11): 2118-2128.
[5] LAI Xiao-bo, ZHU Shi-qiang, FANG Chun-jie. A three-dimensional reconstruction algorithm for complex
background image and its medical applications
[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(11): 2061-2067.
[6] BAO Bi-sai, WU Jian-rong , LOU Xiao-jun, LIU Hai-tao. Feature fusion algorithm based on two-dimensional
feature matrix
[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(11): 2081-2088.