Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2021, Vol. 55 Issue (1): 116-123    DOI: 10.3785/j.issn.1008-973X.2021.01.014
计算机技术、自动控制技术     
基于生成对抗网络的偏转人脸转正
胡惠雅1,2(),盖绍彦1,2,3,达飞鹏1,2,3,*()
1. 东南大学 复杂工程系统测量与控制教育部重点实验室，江苏 南京 210096
2. 东南大学 自动化学院，江苏 南京 210096
3. 东南大学深圳研究院，广东 深圳 518000
Face frontalization based on generative adversarial network
Hui-ya HU1,2(),Shao-yan GAI1,2,3,Fei-peng DA1,2,3,*()
1. Key Laboratory of Measurement and Control of Complex System of Engineering, Ministry of Education, Southeast University, Nanjing 210096, China
2. School of Automation, Southeast University, Nanjing 210096, China
3. Shenzhen Institute of Southeast University, Shenzhen 518000, China
 全文: PDF(1638 KB)   HTML
摘要：

为了提高偏转人脸转正的效果，借鉴双通道生成对抗网络（TP-GAN）双通道生成的思想，将原始网络中的深度卷积生成对抗网络（DCGAN）替换成边界均衡生成对抗网络（BEGAN）. 在传统两者对抗的网络结构中加入判别人脸身份的分类器，形成三者对抗的网络结构. 经实验对比可知，与在生成器损失函数中添加约束相比，结构上加入分类器对人脸身份一致性的保持更加有效. TP-GAN存在训练复杂、模式崩溃等难题，使用BEGAN的网络结构，可以避免这些问题，提高训练效率. 在Multi-PIE数据集及LFW上的实验结果表明，利用提出的方法能够高效地生成高质量的正面人脸图片，且保留人脸的身份特征.
关键词： 人脸生成;  分类器;  模式崩溃;  生成对抗网络（GAN）    
Abstract:

The original deep convolutional generative adversarial network (DCGAN) structure was replaced with boundary equilibrium generative adversarial network (BEGAN) by drawing on the experience of two-pathway generative adversarial network (TP-GAN) in order to improve the effect of frontalizing the rotated faces. The classifier used for discriminating face identity was added to the traditional two player network in order to form the three-player structure. Adding classifier is more effective for maintaining the consistency of face identity compared with adding constraints in the generator loss function. The adopted BEGAN can improve the training efficiency aiming at the problem that TP-GAN are complex to train and prone to the mode collapse. The experimental results on the multiple images across pose, illumination and expression (Multi-PIE) and labeled faces in the wild (LFW) dataset show that the proposed method can generate high-quality frontal face images with retaining the identity information.
Key words: face generation    classifier    mode collapse    generative adversarial network (GAN)
收稿日期: 2020-01-03 出版日期: 2021-01-05
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目（51475092）；深圳市知识创新计划基础研究项目（JCYJ20180306174455080）
通讯作者: 达飞鹏     E-mail: 220171559@seu.edu.cn;dafp@seu.edu.cn
作者简介: 胡惠雅（1996—），女，硕士生，从事计算机视觉的研究. orcid.org/0000-0003-1529-5166. E-mail： 220171559@seu.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
胡惠雅
盖绍彦
达飞鹏

引用本文:

胡惠雅,盖绍彦,达飞鹏. 基于生成对抗网络的偏转人脸转正[J]. 浙江大学学报(工学版), 2021, 55(1): 116-123.

Hui-ya HU,Shao-yan GAI,Fei-peng DA. Face frontalization based on generative adversarial network. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 116-123.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2021.01.014    或    http://www.zjujournals.com/eng/CN/Y2021/V55/I1/116

图 1  整体网络结构示意图
图 2  cGAN结构下单路径人脸合成效果图
图 3  双路径生成正面人脸结构示意图
图 4  判别器结构示意图
图 5  分类器结构示意图
图 6  Multi-PIE人脸集不同角度下的人脸偏转结果
方法 R/%
±90° ±75° ±60° ±45° ±30° ±15°
Hassner[21] ? ? 44.81 74.68 89.59 96.78
TP-GAN[13] 64.03 84.10 92.93 98.58 99.85 99.78
DR-GAN[22] ? ? 83.20 86.20 90.10 94.00
本文方法 66.76 93.20 95.28 98.78 99.83 99.60
表 1  Multi-PIE人脸数据集在Setting-1下的人脸Rank-1识别率
方法 R/%
±90° ±75° ±60° ±45° ±30° ±15°
DR-GAN[22] ? ? 83.20 86.20 90.10 94.00
TP-GAN[13] 64.64 77.43 87.72 95.38 98.06 98.68
FF-GAN[15] 61.20 77.20 85.20 89.70 92.50 94.60
本文方法 70.02 80.60 89.70 95.98 98.72 98.90
表 2  Multi-PIE人脸数据集在Setting-2下的人脸Rank-1识别率
图 7  不同方法在LFW数据集上的生成效果对比图
方法 R/%
±90° ±75° ±60° ±45° ±30° ±15°
w/o ${\rm{D}}$ 46.28 53.44 55.65 57.59 60.10 67.62
w/o ${L_{{\rm{rot}}}}$ 54.65 65.45 80.39 87.05 92.89 94.49
w/o ${L_{{\rm{sym}}}}$ 68.93 78.32 85.20 93.96 95.37 97.58
w/o ${\rm{C}}$ 67.06 76.83 86.73 93.78 96.35 97.06
本文方法 70.02 80.60 89.70 95.98 98.72 98.90
表 3  不同结构模型在Setting-2下的Rank-1识别率对比
图 8  不同结构模型（损失函数不同）的生成效果对比图
1 SUN Y, CHEN Y, WANG X, et al. Deep learning face representation by joint identification-verification [C] // International Conference on Neural Information Processing Systems. Montreal: MIT Press, 2014.
2 SUN Y, WANG X G, TANG X O. Deep learning face representation from predicting 10, 000 classes [C] // IEEE Conference on Computer Vision and Pattern Recognition. Ohio: IEEE, 2014: 1891-1898.
3 LI S, LIU X, CHAI X, et al Maximal likelihood correspondence estimation for face recognition across pose[J]. IEEE Transactions on Image Processing, 2014, 23 (10): 4587- 4600
doi: 10.1109/TIP.2014.2351265
4 WISKOTT L, FELLOUS J M, NORBERT K, et al Face recognition by elastic bunch graph matching[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 19 (7): 456- 463
5 BISWASS S, AGGARWAL G, FLYNN P J, et al Pose-robust recognition of low-resolution face images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (12): 3037- 3049
doi: 10.1109/TPAMI.2013.68
6 CHEN D, CAO X, WEN F, et al. Blessing of dimensionality: high-dimensional feature and its efficient compression for face verification [C]// IEEE Conference on Computer Vision and Pattern Recognition. Portland: IEEE, 2013: 3025-3032.
7 ZHANG Y, SHAO M, WONG E K, et al. Random faces guided sparse many-to-one encoder for pose-invariant face recognition [C]// Proceedings of the 2013 IEEE International Conference on Computer Vision. Sidney: IEEE, 2013: 2416-2423.
8 WU X, HE R, SUN Z, et al A light CNN for deep face representation with noisy labels[J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (11): 2884- 2896
doi: 10.1109/TIFS.2018.2833032
9 SCHROFF F, KALENICHENKO D, PHILBIN J. Facenet: a unified embedding for face recognition and clustering [C]// IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 815-823.
10 PRABHU U, HEO J, SAVVIDES M Unconstrained pose-invariant face recognition using 3D generic elastic models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33 (10): 1952- 1961
doi: 10.1109/TPAMI.2011.123
11 BLANZ V, VETTER T. A morphable model for the synthesis of 3D faces [C]// Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques. Los Angeles: ACM, 1999: 187-194.
12 ALDRIAN O, SMITH W A P, et al Inverse] rendering of faces with a 3D morphable model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (5): 1080- 1093
doi: 10.1109/TPAMI.2012.206
13 HUANG R, SHU Z, TIANYU L, et al. Beyond face rotation: global and local perception GAN for photorealistic and identity preserving frontal view synthesis [C]// IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2439-2448.
14 ZHAO J, CHENG Y, XIU Y, et al. Towards pose invariant face recognition in the wild [C]// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2207-2216.
15 YIN X, YU X, SOHN K, et al. Towards large-pose face frontalization in the wild [C]// IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3990-3999.
16 ZHU X, LEI Z, YAN J J, et al. High-fidelity pose and expression normalization for face recognition in the wild [C]// IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 787-796.
17 SHEN Y J, LUO P, YAN J J, et al. FaceID-GAN: learning a symmetry three-player GAN for identity-preserving face synthesis [C]// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 821-830.
18 BERTHELOT D, SCHUMM T, METZ L. BEGAN: boundary equilibrium generative adversarial networks [EB/OL]. (2017-05-31). https://arxiv.org/pdf/1703.10717.pdf.
19 ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5967–5976.
20 ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN [EB/OL]. (2017-12-06). https://arxiv.org/pdf/1701.07875.pdf.
21 HASSNER T, HAREL S, PAZ, et al. Effective face frontalization in unconstrained images [C]// IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 787-796.
[1] 陈太波,张翠芳. 后验概率图与补白模型二次融合的关键词识别[J]. 浙江大学学报(工学版), 2020, 54(6): 1170-1176.
[2] 刘坤,文熙,黄闽茗,杨欣欣,毛经坤. 基于生成对抗网络的太阳能电池缺陷增强方法[J]. 浙江大学学报(工学版), 2020, 54(4): 684-693.
[3] 胡天磊,王皓波,尹文栋. 基于深度双向分类器链的多标签新闻分类算法[J]. 浙江大学学报(工学版), 2019, 53(11): 2110-2117.
[4] 李滔, 王士同. 增量式0阶TSK模糊分类器及鲁棒改进[J]. 浙江大学学报(工学版), 2017, 51(10): 1901-1911.
[5] 柯海丰,应晶. 基于R-ELM的实时车牌字符识别技术[J]. 浙江大学学报(工学版), 2014, 48(7): 1209-1216.
[6] 柯海丰,应晶. 基于R-ELM的实时车牌字符识别技术[J]. J4, 2014, 48(2): 0-0.
[7] 李文书,何芳芳,钱沄涛,周昌乐. 基于Adaboost-高斯过程分类的人脸表情识别[J]. J4, 2012, 46(1): 79-83.
[8] 罗建宏,陈德钊. 兼顾正确率和差异性的自适应集成算法及应用[J]. J4, 2011, 45(3): 557-562.
[9] 张宝军, 潘雪增, 王界兵, 等. 基于多代理的混合式入侵检测系统模型[J]. J4, 2009, 43(6): 987-993.
[10] 张赞超, 夏顺仁. 全自动尿液图像识别技术[J]. J4, 2009, 43(5): 832-838.
[11] 郑恩辉 许宏 李平 宋执环. 基于ν-SVM的不平衡数据挖掘研究[J]. J4, 2006, 40(10): 1682-1687.
[12] 任俊 李志能. 支持向量机在字符分类识别中的应用[J]. J4, 2005, 39(8): 1136-1141.