Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2022, Vol. 56 Issue (10): 1948-1957    DOI: 10.3785/j.issn.1008-973X.2022.10.006
自动化技术、信息工程     
基于深度卷积和自编码器增强的微表情判别
付晓峰(),牛力
杭州电子科技大学 计算机学院，浙江 杭州 310018
Micro-expression classification based on deep convolution and auto-encoder enhancement
Xiao-feng FU(),Li NIU
School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
 全文: PDF(2816 KB)   HTML
摘要：

为了判别微表情种类，提出基于深度卷积神经网络和迁移学习的微表情种类判别网络MecNet. 为了提高MecNet在CASME II、SMIC和SAMM联合数据库上的微表情种类判别准确率，提出基于自编码器的微表情生成网络MegNet，以扩充训练集. 使用CASME II亚洲人的微表情样本，生成欧美人的微表情样本. 设计卷积结构实现图像编码，设计基于子像素卷积的特征图上采样模块实现图像解码，设计基于图像结构相似性的损失函数用于网络优化. 将生成的欧美人的微表情样本加入MecNet训练集. 实验结果表明，使用MegNet扩充训练集能够有效地提高MecNet微表情种类判别准确率. 结合MegNet、MecNet的算法在CASME II、SMIC和SAMM组成的联合数据库上的表现优于大部分现有算法.
关键词： 微表情种类判别;  深度卷积神经网络;  迁移学习;  自编码器    
Abstract:

A micro-expression classification network named MecNet was proposed based on the deep convolutional neural network and transfer learning in order to classify the types of micro-expressions. MegNet was proposed to expand the training set in order to improve the accuracy of micro-expressions classification of MecNet on the joint database of CASME II, SMIC and SAMM. MegNet is a micro-expression sample generation network based on the auto-encoder. Asian micro-expression samples of CASME II were used to generate western micro-expression samples. A convolution structure was designed to encode images, and a feature map upsampling module was designed based on the sub-pixel convolution to decode images. A loss function based on the structural similarity of images was designed to optimize the network. The generated western micro-expression samples were added to the training set of MecNet. The experimental results show that the accuracy of micro-expression classification of MecNet can?be?effectively?improved?by?using?MegNet?to?expand?training?sets. The algorithm combining MegNet and MecNet performs better than the most existing algorithms on the joint database composed of CASME II, SMIC and SAMM.
Key words: micro-expression classification    deep convolutional neural network    transfer learning    auto-encoder
收稿日期: 2021-10-06 出版日期: 2022-10-25
CLC:  TP 301  
基金资助: 国家自然科学基金资助项目（61672199）
作者简介: 付晓峰（1981—），女，副教授，博士，从事计算机视觉、模式识别、人工智能等研究. orcid.org/0000-0003-4903-5266.E-mail： fuxiaofeng@hdu.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
付晓峰
牛力

引用本文:

付晓峰,牛力. 基于深度卷积和自编码器增强的微表情判别[J]. 浙江大学学报(工学版), 2022, 56(10): 1948-1957.

Xiao-feng FU,Li NIU. Micro-expression classification based on deep convolution and auto-encoder enhancement. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 1948-1957.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2022.10.006    或    https://www.zjujournals.com/eng/CN/Y2022/V56/I10/1948

图 1  传统自编码器的结构图
图 2  微表情生成网络的流程图
图 3  MegNet编码器的结构图
图 4  MegNet解码器的结构图
编码器 解码器
网络层 特征图尺寸 网络层 特征图尺寸
输入层 128×128×3 输入层 16×16×512
5×5×128 Conv 64×64×128 3×3×2 048 Conv 16×16×2 048
5×5×256 Conv 32×32×256 PixelShuffle 32×32×512
5×5×512 Conv 16×16×512 3×3×1 024 Conv 32×32×1 024
5×5×1 024 Conv 8×8×1 024 PixelShuffle 64×64×256
Flatten 65536 3×3×512 Conv 64×64×512
512 FC 512 PixelShuffle 128×128×128
8×8×512 FC 32 768 5×5×3 Conv 128×128×3
变形8×8×512 8×8×512 输出层 128×128×3
3×3×2 048 Conv 8×8×2 048 — —
PixelShuffle 16×16×512 — —
输出层 16×16×512 — —
表 1  MegNet编码器和解码器各层运算之后的特征图尺寸
图 5  子像素卷积神经网络的结构
图 6  MegNet特征图上采样模块的结构
图 7  图像结构相似度测量系统图
图 8  微表情种类判别网络的结构图
图 9  微表情样本生成实验所用的人脸展示
编号 数量 编号 数量
A1 2 220 B1 2 956
A2 1 226 B2 5 323
A3 1 154 B3 7 148
A4 1 141 B4 7 360
A5 1 096 B5 4 191
A6 990 B6 7 611
A7 956 B7 5 501
A8 884 B8 5 312
— — B9 4 911
— — B10 10 801
表 2  集合A和B的个体样本数量
编号 B1 B2 B3 B4 B5 B6 B7 B8 B9 B10
A1 A1B1 — — — — A1B6 A1B7 A1B8 — A1B10
A2 — A2B2 A2B3 — A2B5 — A2B7 — A2B9 —
A3 — A3B2 A3B3 A3B4 A3B5 — — — A3B9 —
A4 A4B1 — — — — — A4B7 A4B8 A4B9 A4B10
A5 A5B1 — — — — A5B6 — A5B8 A5B9 A5B10
A6 A6B1 — A6B3 A6B4 — A6B6 — A6B8 — —
A7 — — A7B3 — — A7B6 A7B7 A7B8 — A7B10
A8 — A8B2 — A8B4 A8B5 A8B6 — — — A8B10
表 3  集合A和B的实验组合
图 10  A1B1实验组训练过程的预览图
图 11  训练预览图详解
图 12  MegNet生成的微表情样本示例
情感类别 SMIC CASME II SAMM 联合数据库
Negative 70 88 92 250
Positive 51 32 26 109
Surprise 43 25 15 83
合计 164 145 133 442
表 4  联合数据库的样本分布
方法 联合数据库 SMIC CASME II SAMM
UF1 UAR UF1 UAR UF1 UAR UF1 UAR
LBP-TOP[18] 0.588 2 0.578 5 0.200 0 0.528 0 0.702 6 0.742 9 0.395 4 0.410 2
Bi-WOOF[19] 0.629 6 0.622 7 0.572 7 0.582 9 0.780 5 0.802 6 0.521 1 0.513 9
OFF-ApexNet[8] 0.719 6 0.709 6 0.681 7 0.669 5 0.876 4 0.868 1 0.540 9 0.539 2
CapsuleNet[20] 0.652 0 0.650 6 0.582 0 0.587 7 0.706 8 0.701 8 0.620 9 0.598 9
Dual-Inception[9] 0.732 2 0.727 8 0.664 5 0.672 6 0.862 1 0.856 0 0.586 8 0.566 3
STSTNet[21] 0.735 3 0.760 5 0.680 1 0.701 3 0.838 2 0.868 6 0.658 8 0.681 0
EMR[22] 0.788 5 0.782 4 0.746 1 0.753 0 0.829 3 0.820 9 0.775 4 0.715 2
LGCcon[5] — — — — 0.792 9 0.763 9 0.524 8 0.495 5
LGCconD[5] — — 0.619 5 0.606 6 0.776 2 0.749 9 0.492 4 0.471 1
MecNet 0.763 2 0.770 4 0.720 1 0.731 9 0.866 7 0.851 0 0.735 8 0.677 2
MegNet+MecNet 0.789 3 0.805 6 0.742 5 0.751 3 0.867 4 0.852 1 0.768 2 0.709 3
表 5  本文方法与现有方法的性能对比
1 付晓峰, 牛力, 胡卓群, 等 基于过渡帧概念训练的微表情检测深度网络[J]. 浙江大学学报: 工学版, 2020, 54 (11): 2128- 2137
FU Xiao-feng, NIU Li, HU Zhuo-qun, et al Deep micro-expression spotting network training based on concept of transition frame[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (11): 2128- 2137
2 BEN X, REN Y, ZHANG J, et al Video-based facial micro-expression analysis: a survey of datasets, features and algorithms[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (9): 5826- 5846
3 WEI J, LU G, YAN J A comparative study on movement feature in different directions for micro-expression recognition[J]. Neurocomputing, 2021, 449: 159- 171
doi: 10.1016/j.neucom.2021.03.063
4 ZHAO S, TAO H, ZHANG Y, et al A two-stage 3D CNN based learning method for spontaneous micro-expression recognition[J]. Neurocomputing, 2021, 448: 276- 289
doi: 10.1016/j.neucom.2021.03.058
5 LI Y, HUANG X, ZHAO G Joint local and global information learning with single apex frame detection for micro-expression recognition[J]. IEEE Transactions on Image Processing, 2021, 30: 249- 263
doi: 10.1109/TIP.2020.3035042
6 AOUAYEB M, HAMIDOUCHE W, SOLADIE C, et al Micro-expression recognition from local facial regions[J]. Signal Processing: Image Communication, 2021, 99 (116457): 1- 9
7 LIU K, JIN Q, XU H, et al Micro-expression recognition using advanced genetic algorithm[J]. Signal Processing: Image Communication, 2021, 93 (116153): 1- 10
8 GAN Y S, LIONG S T, YAU W C, et al OFF-ApexNet on micro-expression recognition system[J]. Signal Processing: Image Communication, 2019, 74: 129- 139
doi: 10.1016/j.image.2019.02.005
9 ZHOU L, MAO Q, XUE L. Dual-Inception network for cross-database micro-expression recognition [C]// 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE, 2019: 1-5.
10 YAN W J, LI X, WANG S J, et al CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLOS ONE, 2014, 9 (1): 1- 8
11 LI X, PFISTER T, HUANG X, et al. A spontaneous micro-expression database: inducement, collection and baseline [C]// 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai: IEEE, 2013: 1–6.
12 DAVISON A K, LANSLEY C, COSTEN N, et al SAMM: a spontaneous micro-facial movement dataset[J]. IEEE Transactions on Affective Computing, 2018, 9 (1): 116- 129
doi: 10.1109/TAFFC.2016.2573832
13 HUANG S W, LIN C T, CHEN S P, et al. AugGAN: cross domain adaptation with GAN-based data augmentation [C]// European Conference on Computer Vision. Munich: Springer, 2018: 731-744.
14 SHI W, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
15 SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GANs [C]// 30th Conference on Neural Information Processing Systems. Barcelona: [s. n.], 2016: 1-9.
16 WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 1- 14
doi: 10.1109/TIP.2004.827769
17 SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, inception-resnet and the impact of residual connections on learning [C]// 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2017: 4-12.
18 ZHAO G, PIETIKAINEN M Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29 (6): 915- 928
doi: 10.1109/TPAMI.2007.1110
19 LIONG S T, SEE J, WONG K, et al Less is more: micro-expression recognition from video using apex frame[J]. Signal Processing: Image Communication, 2018, 62: 82- 92
doi: 10.1016/j.image.2017.11.006
20 QUANG N V, CHUN J, TOKUYAMA T. CapsuleNet for micro-expression recognition [C]// 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE, 2019: 1-7.
21 LIONG S T, GAN Y, SEE J, et al. Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition [C]// 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE, 2019: 1-5.
[1] 潘裕斌,王华,陈捷,洪荣晶. 基于核极限学习机自编码器的转盘轴承寿命状态识别[J]. 浙江大学学报(工学版), 2022, 56(9): 1856-1866.
[2] 华夏,王新晴,芮挺,邵发明,王东. 视觉感知的无人机端到端目标跟踪控制技术[J]. 浙江大学学报(工学版), 2022, 56(7): 1464-1472.
[3] 高一聪,王彦坤,费少梅,林琼. 基于迁移学习的机械制图智能评阅方法[J]. 浙江大学学报(工学版), 2022, 56(5): 856-863, 889.
[4] 白文超,韩希先,王金宝. 基于条件生成模型的高效近似查询处理框架[J]. 浙江大学学报(工学版), 2022, 56(5): 995-1005.
[5] 刘兴,余建波. 注意力卷积GRU自编码器及其在工业过程监控的应用[J]. 浙江大学学报(工学版), 2021, 55(9): 1643-1651.
[6] 陈智超,焦海宁,杨杰,曾华福. 基于改进MobileNet v2的垃圾图像分类算法[J]. 浙江大学学报(工学版), 2021, 55(8): 1490-1499.
[7] 郑英杰,吴松荣,韦若禹,涂振威,廖进,刘东. 基于目标图像FCM算法的地铁定位点匹配及误报排除方法[J]. 浙江大学学报(工学版), 2021, 55(3): 586-593.
[8] 许昱晖,舒俊清,宋亚,郑宇,夏唐斌. 基于多时间尺度相似性的涡扇发动机寿命预测[J]. 浙江大学学报(工学版), 2021, 55(10): 1937-1947.
[9] 康庄,杨杰,郭濠奇. 基于机器视觉的垃圾自动分类系统设计[J]. 浙江大学学报(工学版), 2020, 54(7): 1272-1280.
[10] 沈宗礼,余建波. 基于迁移学习与深度森林的晶圆图缺陷识别[J]. 浙江大学学报(工学版), 2020, 54(6): 1228-1239.
[11] 江金生,任浩然,李瀚野. 基于卷积自编码器的地震数据处理[J]. 浙江大学学报(工学版), 2020, 54(5): 978-984.
[12] 付晓峰,牛力,胡卓群,李建军,吴卿. 基于过渡帧概念训练的微表情检测深度网络[J]. 浙江大学学报(工学版), 2020, 54(11): 2128-2137.
[13] 王万良,杨小涵,赵燕伟,高楠,吕闯,张兆娟. 采用卷积自编码器网络的图像增强算法[J]. 浙江大学学报(工学版), 2019, 53(9): 1728-1740.