Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2021, Vol. 55 Issue (1): 116-123    DOI: 10.3785/j.issn.1008-973X.2021.01.014
    
Face frontalization based on generative adversarial network
Hui-ya HU1,2(),Shao-yan GAI1,2,3,Fei-peng DA1,2,3,*()
1. Key Laboratory of Measurement and Control of Complex System of Engineering, Ministry of Education, Southeast University, Nanjing 210096, China
2. School of Automation, Southeast University, Nanjing 210096, China
3. Shenzhen Institute of Southeast University, Shenzhen 518000, China
Download: HTML     PDF(1638KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

The original deep convolutional generative adversarial network (DCGAN) structure was replaced with boundary equilibrium generative adversarial network (BEGAN) by drawing on the experience of two-pathway generative adversarial network (TP-GAN) in order to improve the effect of frontalizing the rotated faces. The classifier used for discriminating face identity was added to the traditional two player network in order to form the three-player structure. Adding classifier is more effective for maintaining the consistency of face identity compared with adding constraints in the generator loss function. The adopted BEGAN can improve the training efficiency aiming at the problem that TP-GAN are complex to train and prone to the mode collapse. The experimental results on the multiple images across pose, illumination and expression (Multi-PIE) and labeled faces in the wild (LFW) dataset show that the proposed method can generate high-quality frontal face images with retaining the identity information.



Key wordsface generation      classifier      mode collapse      generative adversarial network (GAN)     
Received: 03 January 2020      Published: 05 January 2021
CLC:  TP 391  
Corresponding Authors: Fei-peng DA     E-mail: 220171559@seu.edu.cn;dafp@seu.edu.cn
Cite this article:

Hui-ya HU,Shao-yan GAI,Fei-peng DA. Face frontalization based on generative adversarial network. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 116-123.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2021.01.014     OR     http://www.zjujournals.com/eng/Y2021/V55/I1/116


基于生成对抗网络的偏转人脸转正

为了提高偏转人脸转正的效果,借鉴双通道生成对抗网络(TP-GAN)双通道生成的思想,将原始网络中的深度卷积生成对抗网络(DCGAN)替换成边界均衡生成对抗网络(BEGAN). 在传统两者对抗的网络结构中加入判别人脸身份的分类器,形成三者对抗的网络结构. 经实验对比可知,与在生成器损失函数中添加约束相比,结构上加入分类器对人脸身份一致性的保持更加有效. TP-GAN存在训练复杂、模式崩溃等难题,使用BEGAN的网络结构,可以避免这些问题,提高训练效率. 在Multi-PIE数据集及LFW上的实验结果表明,利用提出的方法能够高效地生成高质量的正面人脸图片,且保留人脸的身份特征.


关键词: 人脸生成,  分类器,  模式崩溃,  生成对抗网络(GAN) 
Fig.1 Diagram of whole network structure
Fig.2 Results of single-path generator by cGAN
Fig.3 Diagram of two-pathway generator for generating frontalized faces
Fig.4 Diagram of discriminator structure
Fig.5 Diagram of classifier structure
Fig.6 Synthesis results of face frontalization under varied poses on Multi-PIE dataset
方法 R/%
±90° ±75° ±60° ±45° ±30° ±15°
Hassner[21] ? ? 44.81 74.68 89.59 96.78
TP-GAN[13] 64.03 84.10 92.93 98.58 99.85 99.78
DR-GAN[22] ? ? 83.20 86.20 90.10 94.00
本文方法 66.76 93.20 95.28 98.78 99.83 99.60
Tab.1 Rank-1 recognition rates on Multi-PIE dataset under Setting-1
方法 R/%
±90° ±75° ±60° ±45° ±30° ±15°
DR-GAN[22] ? ? 83.20 86.20 90.10 94.00
TP-GAN[13] 64.64 77.43 87.72 95.38 98.06 98.68
FF-GAN[15] 61.20 77.20 85.20 89.70 92.50 94.60
本文方法 70.02 80.60 89.70 95.98 98.72 98.90
Tab.2 Rank-1 recognition rates on Multi-PIE dataset under Setting-2
Fig.7 Synthesis results of different methods on LFW dataset
方法 R/%
±90° ±75° ±60° ±45° ±30° ±15°
w/o ${\rm{D}}$ 46.28 53.44 55.65 57.59 60.10 67.62
w/o ${L_{{\rm{rot}}}}$ 54.65 65.45 80.39 87.05 92.89 94.49
w/o ${L_{{\rm{sym}}}}$ 68.93 78.32 85.20 93.96 95.37 97.58
w/o ${\rm{C}}$ 67.06 76.83 86.73 93.78 96.35 97.06
本文方法 70.02 80.60 89.70 95.98 98.72 98.90
Tab.3 Rank-1 recognition rates of different model under Setting-2
Fig.8 Model comparison:synthesis results using different loss function
[1]   SUN Y, CHEN Y, WANG X, et al. Deep learning face representation by joint identification-verification [C] // International Conference on Neural Information Processing Systems. Montreal: MIT Press, 2014.
[2]   SUN Y, WANG X G, TANG X O. Deep learning face representation from predicting 10, 000 classes [C] // IEEE Conference on Computer Vision and Pattern Recognition. Ohio: IEEE, 2014: 1891-1898.
[3]   LI S, LIU X, CHAI X, et al Maximal likelihood correspondence estimation for face recognition across pose[J]. IEEE Transactions on Image Processing, 2014, 23 (10): 4587- 4600
doi: 10.1109/TIP.2014.2351265
[4]   WISKOTT L, FELLOUS J M, NORBERT K, et al Face recognition by elastic bunch graph matching[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 19 (7): 456- 463
[5]   BISWASS S, AGGARWAL G, FLYNN P J, et al Pose-robust recognition of low-resolution face images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (12): 3037- 3049
doi: 10.1109/TPAMI.2013.68
[6]   CHEN D, CAO X, WEN F, et al. Blessing of dimensionality: high-dimensional feature and its efficient compression for face verification [C]// IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE, 2013: 3025-3032.
[7]   ZHANG Y, SHAO M, WONG E K, et al. Random faces guided sparse many-to-one encoder for pose-invariant face recognition [C]// Proceedings of the 2013 IEEE International Conference on Computer Vision. Sidney: IEEE, 2013: 2416-2423.
[8]   WU X, HE R, SUN Z, et al A light CNN for deep face representation with noisy labels[J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (11): 2884- 2896
doi: 10.1109/TIFS.2018.2833032
[9]   SCHROFF F, KALENICHENKO D, PHILBIN J. Facenet: a unified embedding for face recognition and clustering [C]// IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 815-823.
[10]   PRABHU U, HEO J, SAVVIDES M Unconstrained pose-invariant face recognition using 3D generic elastic models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33 (10): 1952- 1961
doi: 10.1109/TPAMI.2011.123
[11]   BLANZ V, VETTER T. A morphable model for the synthesis of 3D faces [C]// Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques. Los Angeles: ACM, 1999: 187-194.
[12]   ALDRIAN O, SMITH W A P, et al Inverse] rendering of faces with a 3D morphable model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (5): 1080- 1093
doi: 10.1109/TPAMI.2012.206
[13]   HUANG R, SHU Z, TIANYU L, et al. Beyond face rotation: global and local perception GAN for photorealistic and identity preserving frontal view synthesis [C]// IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2439-2448.
[14]   ZHAO J, CHENG Y, XIU Y, et al. Towards pose invariant face recognition in the wild [C]// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2207-2216.
[15]   YIN X, YU X, SOHN K, et al. Towards large-pose face frontalization in the wild [C]// IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3990-3999.
[16]   ZHU X, LEI Z, YAN J J, et al. High-fidelity pose and expression normalization for face recognition in the wild [C]// IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 787-796.
[17]   SHEN Y J, LUO P, YAN J J, et al. FaceID-GAN: learning a symmetry three-player GAN for identity-preserving face synthesis [C]// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 821-830.
[18]   BERTHELOT D, SCHUMM T, METZ L. BEGAN: boundary equilibrium generative adversarial networks [EB/OL]. (2017-05-31). https://arxiv.org/pdf/1703.10717.pdf.
[19]   ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5967–5976.
[20]   ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN [EB/OL]. (2017-12-06). https://arxiv.org/pdf/1701.07875.pdf.
[21]   HASSNER T, HAREL S, PAZ, et al. Effective face frontalization in unconstrained images [C]// IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 787-796.
[1] Tai-bo CHEN,Cui-fang ZHANG. Keyword recognition based on twice fusion of Posteriorgram and filler model[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(6): 1170-1176.
[2] Kun LIU,Xi WEN,Min-ming HUANG,Xin-xin YANG,Jing-kun MAO. Solar cell defect enhancement method based on generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(4): 684-693.
[3] Tian-lei HU,Hao-bo WANG,Wen-dong YIN. Multi-label news classification algorithm based on deep bi-directional classifier chains[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(11): 2110-2117.
[4] LI Tao, WANG Shi-tong. Incremental zero-order TSK fuzzy classifier and its robust version[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(10): 1901-1911.
[5] KE Hai-feng, YING Jing. Real-time license character recognition technology based on R-ELM[J]. Journal of ZheJiang University (Engineering Science), 2014, 48(7): 1209-1216.
[6] KE Hai-feng, YING Jing. Real-time license character recognition technology based on R-ELM[J]. Journal of ZheJiang University (Engineering Science), 2014, 48(2): 0-0.
[7] LI Wen-shu, HE Fang-fang, QIAN Yun-tao, ZHOU Chang-le. Facial expression recognition based on
Adaboost-Gaussian process classification
[J]. Journal of ZheJiang University (Engineering Science), 2012, 46(1): 79-83.
[8] LUO Jian-hong, CHEN De-zhao. Application of adaptive ensemble algorithm based on
correctness and diversity
[J]. Journal of ZheJiang University (Engineering Science), 2011, 45(3): 557-562.
[9] ZHANG Bao-Jun, BO Xue-Ceng, WANG Jie-Bing, et al. Multi-agent based hybrid Intrusion detection system[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(6): 987-993.
[10] ZHANG Zan-Chao, JIA Shun-Ren. Automatic urinary image processing[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(5): 832-838.