Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (10): 1948-1957    DOI: 10.3785/j.issn.1008-973X.2022.10.006
    
Micro-expression classification based on deep convolution and auto-encoder enhancement
Xiao-feng FU(),Li NIU
School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
Download: HTML     PDF(2816KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A micro-expression classification network named MecNet was proposed based on the deep convolutional neural network and transfer learning in order to classify the types of micro-expressions. MegNet was proposed to expand the training set in order to improve the accuracy of micro-expressions classification of MecNet on the joint database of CASME II, SMIC and SAMM. MegNet is a micro-expression sample generation network based on the auto-encoder. Asian micro-expression samples of CASME II were used to generate western micro-expression samples. A convolution structure was designed to encode images, and a feature map upsampling module was designed based on the sub-pixel convolution to decode images. A loss function based on the structural similarity of images was designed to optimize the network. The generated western micro-expression samples were added to the training set of MecNet. The experimental results show that the accuracy of micro-expression classification of MecNet can?be?effectively?improved?by?using?MegNet?to?expand?training?sets. The algorithm combining MegNet and MecNet performs better than the most existing algorithms on the joint database composed of CASME II, SMIC and SAMM.

Key words： micro-expression classification      deep convolutional neural network      transfer learning      auto-encoder     
Received: 06 October 2021      Published: 25 October 2022
CLC:  TP 301  
Fund:  国家自然科学基金资助项目（61672199）
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Xiao-feng FU
Li NIU
Cite this article:

Xiao-feng FU,Li NIU. Micro-expression classification based on deep convolution and auto-encoder enhancement. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 1948-1957.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.10.006     OR     https://www.zjujournals.com/eng/Y2022/V56/I10/1948


基于深度卷积和自编码器增强的微表情判别

为了判别微表情种类，提出基于深度卷积神经网络和迁移学习的微表情种类判别网络MecNet. 为了提高MecNet在CASME II、SMIC和SAMM联合数据库上的微表情种类判别准确率，提出基于自编码器的微表情生成网络MegNet，以扩充训练集. 使用CASME II亚洲人的微表情样本，生成欧美人的微表情样本. 设计卷积结构实现图像编码，设计基于子像素卷积的特征图上采样模块实现图像解码，设计基于图像结构相似性的损失函数用于网络优化. 将生成的欧美人的微表情样本加入MecNet训练集. 实验结果表明，使用MegNet扩充训练集能够有效地提高MecNet微表情种类判别准确率. 结合MegNet、MecNet的算法在CASME II、SMIC和SAMM组成的联合数据库上的表现优于大部分现有算法.


关键词： 微表情种类判别,  深度卷积神经网络,  迁移学习,  自编码器 
Fig.1 Structure diagram of traditional auto-encoder
Fig.2 Flow chart of micro-expression generation network
Fig.3 Architecture of MegNet encoder
Fig.4 Structure diagram of MegNet decoder
编码器 解码器
网络层 特征图尺寸 网络层 特征图尺寸
输入层 128×128×3 输入层 16×16×512
5×5×128 Conv 64×64×128 3×3×2 048 Conv 16×16×2 048
5×5×256 Conv 32×32×256 PixelShuffle 32×32×512
5×5×512 Conv 16×16×512 3×3×1 024 Conv 32×32×1 024
5×5×1 024 Conv 8×8×1 024 PixelShuffle 64×64×256
Flatten 65536 3×3×512 Conv 64×64×512
512 FC 512 PixelShuffle 128×128×128
8×8×512 FC 32 768 5×5×3 Conv 128×128×3
变形8×8×512 8×8×512 输出层 128×128×3
3×3×2 048 Conv 8×8×2 048 — —
PixelShuffle 16×16×512 — —
输出层 16×16×512 — —
Tab.1 Feature map size of MegNet encoder and decoder after each layer operation
Fig.5 Architecture of subpixel convolutional neural network
Fig.6 Architecture of upsampling module on MegNet feature map
Fig.7 Diagram of image structure similarity measurement system
Fig.8 Architecture of micro-expression classification network
Fig.9 Face display for micro-expression sample generation experiment
编号 数量 编号 数量
A1 2 220 B1 2 956
A2 1 226 B2 5 323
A3 1 154 B3 7 148
A4 1 141 B4 7 360
A5 1 096 B5 4 191
A6 990 B6 7 611
A7 956 B7 5 501
A8 884 B8 5 312
— — B9 4 911
— — B10 10 801
Tab.2 Number of individual samples of sets A and B
编号 B1 B2 B3 B4 B5 B6 B7 B8 B9 B10
A1 A1B1 — — — — A1B6 A1B7 A1B8 — A1B10
A2 — A2B2 A2B3 — A2B5 — A2B7 — A2B9 —
A3 — A3B2 A3B3 A3B4 A3B5 — — — A3B9 —
A4 A4B1 — — — — — A4B7 A4B8 A4B9 A4B10
A5 A5B1 — — — — A5B6 — A5B8 A5B9 A5B10
A6 A6B1 — A6B3 A6B4 — A6B6 — A6B8 — —
A7 — — A7B3 — — A7B6 A7B7 A7B8 — A7B10
A8 — A8B2 — A8B4 A8B5 A8B6 — — — A8B10
Tab.3 Experimental combination of sets A and B
Fig.10 Preview of training process of experimental group A1B1
Fig.11 Details of training preview
Fig.12 Example of micro-expressions generated by MegNet
情感类别 SMIC CASME II SAMM 联合数据库
Negative 70 88 92 250
Positive 51 32 26 109
Surprise 43 25 15 83
合计 164 145 133 442
Tab.4 Sample distribution of joint database
方法 联合数据库 SMIC CASME II SAMM
UF1 UAR UF1 UAR UF1 UAR UF1 UAR
LBP-TOP[18] 0.588 2 0.578 5 0.200 0 0.528 0 0.702 6 0.742 9 0.395 4 0.410 2
Bi-WOOF[19] 0.629 6 0.622 7 0.572 7 0.582 9 0.780 5 0.802 6 0.521 1 0.513 9
OFF-ApexNet[8] 0.719 6 0.709 6 0.681 7 0.669 5 0.876 4 0.868 1 0.540 9 0.539 2
CapsuleNet[20] 0.652 0 0.650 6 0.582 0 0.587 7 0.706 8 0.701 8 0.620 9 0.598 9
Dual-Inception[9] 0.732 2 0.727 8 0.664 5 0.672 6 0.862 1 0.856 0 0.586 8 0.566 3
STSTNet[21] 0.735 3 0.760 5 0.680 1 0.701 3 0.838 2 0.868 6 0.658 8 0.681 0
EMR[22] 0.788 5 0.782 4 0.746 1 0.753 0 0.829 3 0.820 9 0.775 4 0.715 2
LGCcon[5] — — — — 0.792 9 0.763 9 0.524 8 0.495 5
LGCconD[5] — — 0.619 5 0.606 6 0.776 2 0.749 9 0.492 4 0.471 1
MecNet 0.763 2 0.770 4 0.720 1 0.731 9 0.866 7 0.851 0 0.735 8 0.677 2
MegNet+MecNet 0.789 3 0.805 6 0.742 5 0.751 3 0.867 4 0.852 1 0.768 2 0.709 3
Tab.5 Comparison of proposed method with existing methods
[1]   付晓峰, 牛力, 胡卓群, 等 基于过渡帧概念训练的微表情检测深度网络[J]. 浙江大学学报: 工学版, 2020, 54 (11): 2128- 2137
FU Xiao-feng, NIU Li, HU Zhuo-qun, et al Deep micro-expression spotting network training based on concept of transition frame[J]. Journal of Zhejiang University: Engineering Science, 2020, 54 (11): 2128- 2137
[2]   BEN X, REN Y, ZHANG J, et al Video-based facial micro-expression analysis: a survey of datasets, features and algorithms[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (9): 5826- 5846
[3]   WEI J, LU G, YAN J A comparative study on movement feature in different directions for micro-expression recognition[J]. Neurocomputing, 2021, 449: 159- 171
doi: 10.1016/j.neucom.2021.03.063
[4]   ZHAO S, TAO H, ZHANG Y, et al A two-stage 3D CNN based learning method for spontaneous micro-expression recognition[J]. Neurocomputing, 2021, 448: 276- 289
doi: 10.1016/j.neucom.2021.03.058
[5]   LI Y, HUANG X, ZHAO G Joint local and global information learning with single apex frame detection for micro-expression recognition[J]. IEEE Transactions on Image Processing, 2021, 30: 249- 263
doi: 10.1109/TIP.2020.3035042
[6]   AOUAYEB M, HAMIDOUCHE W, SOLADIE C, et al Micro-expression recognition from local facial regions[J]. Signal Processing: Image Communication, 2021, 99 (116457): 1- 9
[7]   LIU K, JIN Q, XU H, et al Micro-expression recognition using advanced genetic algorithm[J]. Signal Processing: Image Communication, 2021, 93 (116153): 1- 10
[8]   GAN Y S, LIONG S T, YAU W C, et al OFF-ApexNet on micro-expression recognition system[J]. Signal Processing: Image Communication, 2019, 74: 129- 139
doi: 10.1016/j.image.2019.02.005
[9]   ZHOU L, MAO Q, XUE L. Dual-Inception network for cross-database micro-expression recognition [C]// 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE, 2019: 1-5.
[10]   YAN W J, LI X, WANG S J, et al CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLOS ONE, 2014, 9 (1): 1- 8
[11]   LI X, PFISTER T, HUANG X, et al. A spontaneous micro-expression database: inducement, collection and baseline [C]// 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai: IEEE, 2013: 1–6.
[12]   DAVISON A K, LANSLEY C, COSTEN N, et al SAMM: a spontaneous micro-facial movement dataset[J]. IEEE Transactions on Affective Computing, 2018, 9 (1): 116- 129
doi: 10.1109/TAFFC.2016.2573832
[13]   HUANG S W, LIN C T, CHEN S P, et al. AugGAN: cross domain adaptation with GAN-based data augmentation [C]// European Conference on Computer Vision. Munich: Springer, 2018: 731-744.
[14]   SHI W, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
[15]   SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GANs [C]// 30th Conference on Neural Information Processing Systems. Barcelona: [s. n.], 2016: 1-9.
[16]   WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 1- 14
doi: 10.1109/TIP.2004.827769
[17]   SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, inception-resnet and the impact of residual connections on learning [C]// 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2017: 4-12.
[18]   ZHAO G, PIETIKAINEN M Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29 (6): 915- 928
doi: 10.1109/TPAMI.2007.1110
[19]   LIONG S T, SEE J, WONG K, et al Less is more: micro-expression recognition from video using apex frame[J]. Signal Processing: Image Communication, 2018, 62: 82- 92
doi: 10.1016/j.image.2017.11.006
[20]   QUANG N V, CHUN J, TOKUYAMA T. CapsuleNet for micro-expression recognition [C]// 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE, 2019: 1-7.
[21]   LIONG S T, GAN Y, SEE J, et al. Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition [C]// 14th IEEE International Conference on Automatic Face and Gesture Recognition. Lille: IEEE, 2019: 1-5.
[1] Yu-bin PAN,Hua WANG,Jie CHEN,Rong-jing HONG. Life condition recognition of slewing bearing based on kernel extreme learning machine based auto-encoder[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(9): 1856-1866.
[2] Xia HUA,Xin-qing WANG,Ting RUI,Fa-ming SHAO,Dong WANG. Vision-driven end-to-end maneuvering object tracking of UAV[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1464-1472.
[3] Wen-chao BAI,Xi-xian HAN,Jin-bao WANG. Efficient approximate query processing framework based on conditional generative model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 995-1005.
[4] Yi-cong GAO,Yan-kun WANG,Shao-mei FEI,Qiong LIN. Intelligent proofreading method of engineering drawing based on transfer learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(5): 856-863, 889.
[5] Zhi-chao CHEN,Hai-ning JIAO,Jie YANG,Hua-fu ZENG. Garbage image classification algorithm based on improved MobileNet v2[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1490-1499.
[6] Ying-jie ZHENG,Song-rong WU,Ruo-yu WEI,Zhen-wei TU,Jin LIAO,Dong LIU. Metro location point matching and false alarm elimination based on FCM algorithm of target image[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 586-593.
[7] Zhuang KANG,Jie YANG,Hao-qi GUO. Automatic garbage classification system based on machine vision[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1272-1280.
[8] Zong-li SHEN,Jian-bo YU. Wafer map defect recognition based on transfer learning and deep forest[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(6): 1228-1239.
[9] Xue-feng ZHANG,Xiu-li CHEN,De-wen SENG. Top-N recommendation algorithm combining user trust and influence[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(2): 311-319.
[10] Xiao-feng FU,Li NIU,Zhuo-qun HU,Jian-jun LI,Qing WU. Deep micro-expression spotting network training based on concept of transition frame[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(11): 2128-2137.
[11] Wan-liang WANG,Xiao-han YANG,Yan-wei ZHAO,Nan GAO,Chuang LV,Zhao-juan ZHANG. Image enhancement algorithm with convolutional auto-encoder network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(9): 1728-1740.
[12] Ting-ting ZHAO,zhe WANG,Yi-nan LU. Heterogeneous information network representation learning based on transition probability matrix (HINtpm)[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(3): 548-554.