Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (7): 1287-1296    DOI: 10.3785/j.issn.1008-973X.2023.07.003
    
Driver fatigue state detection method based on multi-feature fusion
Hao-jie FANG1(),Hong-zhao DONG1,*(),Shao-xuan LIN1,Jian-yu LUO2,Yong FANG2
1. ITS Joint Research Institute, Zhejiang University of Technology, Hangzhou 310014, China
2. Hangzhou Jintong Technology Group Limited Company, Hangzhou 310014, China
Download: HTML     PDF(1481KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

The improved YOLOv5 object detection algorithm was used to detect the facial region of the driver and a multi-feature fusion fatigue state detection method was established aiming at the problem that existing fatigue state detection method cannot be applied to drivers under the epidemic prevention and control. The image tag data including the situation of wearing a mask and the situation without wearing a mask were established according to the characteristics of bus driving. The detection accuracy of eyes, mouth and face regions was improved by increasing the feature sampling times of YOLOv5 model. The BiFPN network structure was used to retain multi-scale feature information, which makes the prediction network more sensitive to targets of different sizes and improves the detection ability of the overall model. A parameter compensation mechanism was proposed combined with face keypoint algorithm in order to improve the accuracy of blink and yawn frame number. A variety of fatigue parameters were fused and normalized to conduct fatigue classification. The results of the public dataset NTHU and the self-made dataset show that the proposed method can recognize the blink and yawn of drivers both with and without masks, and can accurately judge the fatigue state of drivers.

Key words： driver safety      fatigue detection      YOLOv5      video analytics      driving simulation     
Received: 06 August 2022      Published: 17 July 2023
CLC:  TP 391  
Fund:  国家自然科学基金资助项目（61773347）；浙江公益技术研究资助项目（LGF19F030001）
Corresponding Authors: Hong-zhao DONG     E-mail: 1047677234@qq.com;its@zjut.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Hao-jie FANG
Hong-zhao DONG
Shao-xuan LIN
Jian-yu LUO
Yong FANG
Cite this article:

Hao-jie FANG,Hong-zhao DONG,Shao-xuan LIN,Jian-yu LUO,Yong FANG. Driver fatigue state detection method based on multi-feature fusion. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1287-1296.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.07.003     OR     https://www.zjujournals.com/eng/Y2023/V57/I7/1287


多特征融合的驾驶员疲劳状态检测方法

针对现有疲劳状态检测方法无法适用于疫情防控下的驾驶员，利用改进后的YOLOv5目标检测算法，对驾驶员的面部区域进行检测，建立多特征融合的疲劳状态检测方法. 针对公交驾驶特性，建立包含佩戴口罩和未佩戴口罩情况的图像标签数据. 通过增加YOLOv5模型的特征采样次数，提高眼、嘴、面部区域的检测精度. 利用BiFPN网络结构保留多尺度的特征信息，使得预测网络对不同大小的目标更敏感，提升整体模型的检测能力. 结合人脸关键点算法提出参数补偿机制，提高眨眼、打哈欠帧数的准确率. 将多种疲劳参数融合归一化处理，开展疲劳等级划分. 公开数据集NTHU和自制数据集的验证结果表明，该方法对佩戴口罩和未佩戴口罩情况均可以进行眨眼、打哈欠识别，可以准确地判断驾驶员的疲劳状态.


关键词： 驾驶安全,  疲劳检测,  YOLOv5,  视频分析,  模拟驾驶 
Fig.1 Improved YOLOv5 network structure
Fig.2 Schematic of image annotation
Fig.3 Diagram of face key point position
Fig.4 Partial video images of NTHU dataset
Fig.5 Sketch of self-made data set image
模型 M/106 FLOPs/109 AP mAP/%
Um Ym Ce Oe Cm Om Ya
Faster-RCNN [21] 137.1 370.4 99.8 100 73.1 74.2 97.2 98.2 90.5 90.6
YOLOv3-tiny [22] 8.7 13.0 99.0 99.0 90.3 91.0 94.1 94.3 94.0 94.5
SSD[23] 26.3 62.8 99.5 99.8 84.2 90.0 89.8 95.4 98.0 93.8
YOLOv5-S[24] 7.0 15.9 99.0 99.3 91.8 92.5 94.5 95.0 94.5 95.2
YOLOv5-S+Detection layer 8.2 28.3 98.7 99.1 94.8 95.7 97.3 98.8 97.6 97.5
YOLOv5-S+BiFPN 7.1 16.6 99.2 99.4 93.2 94.4 96.3 97.3 98.0 96.8
YOLOv5-S+ Detection layer+BiFPN (本文模型) 9.3 29.9 99.5 99.5 98.6 99.5 99.5 99.5 99.5 99.4
Tab.1 Performance comparison of different models
Fig.6 Checking effect chart of improved model
$ {N_{\rm{e}}} $ $ {N_{{\rm{Pe}}}} $ $ {N_{{\rm{Te}}}} $ $ {R_{\rm{e}}} $
100 436 325 1.34
500 2612 1988 1.31
1000 5223 3962 1.32
Tab.2 Statistics of closed-eye frames
$ {N_{\rm{m}}} $ $ {N_{{\rm{Pm}}}} $ $ {N_{{\rm{Tm}}}} $ $ {R_{\rm{m}}} $
20 1832 1512 1.21
40 3756 3109 1.21
100 9521 7826 1.22
Tab.3 Statistics of yawning frames
人员编号 戴口罩 $ {N_{\rm{e}}} $ $ {N_{{\rm{De}}}} $ $ {N_{{\rm{Ae}}}} $ $ {N_{{\rm{Fe}}}} $ $ {N_{{\rm{Le}}}} $ $ {A_{\rm{e}}} $/%
1 是 512 515 507 8 5 97.5
2 是 474 480 474 6 0 98.8
3 是 624 647 620 27 4 95.2
4 否 681 690 671 19 10 95.8
5 否 563 572 558 14 5 96.7
6 否 415 426 413 13 2 96.5
Tab.4 Blink count analysis
人员编号 戴口罩 $ {N_{\rm{m}}} $ $ {N_{{\rm{Dm}}}} $ $ {N_{{\rm{Am}}}} $ $ {N_{{\rm{Fm}}}} $ $ {N_{{\rm{Lm}}}} $ $ {A_{\rm{m}}} $/%
1 是 6 6 6 0 0 100
2 是 1 1 1 0 0 100
3 是 7 8 7 1 0 87.5
4 否 11 10 10 0 1 90.1
5 否 5 5 5 0 0 100
6 否 3 3 3 0 0 100
Tab.5 Yawning count analysis
疲劳参数 $ {P_{\min }} $ $ {P_{\max }} $ W
BF 13 34 0.1
ECN 0 5 0.2
ECR 0.02 0.06 0.3
YF 0 3 0.2
MOR 0 0.2 0.2
Tab.6 Maximum value and weight of comprehensive fatigue parameters
疲劳状态 Q
清醒 Q < 0.3
疲劳 0.3 ≤ Q < 0.7
重度疲劳 Q ≥ 0.7
Tab.7 Relationship between comprehensive fatigue index and fatigue grade
组数 真实状态 眼部 嘴部 Q 识别结果
BF ECN ECR YF MOR
1 清醒 23 0 0.022 1 0.05 0.18 清醒
1 疲劳 25 3 0.053 2 0.1 0.65 疲劳
1 重度疲劳 34 4 0.050 2 0.12 0.74 重度疲劳
2 清醒 25 1 0.035 0 0 0.21 清醒
2 疲劳 20 2 0.047 3 0.15 0.62 疲劳
2 重度疲劳 20 5 0.055 3 0.2 0.89 重度疲劳
3 清醒 22 0 0.042 0 0 0.21 清醒
3 疲劳 17 1 0.039 3 0.17 0.57 疲劳
3 重度疲劳 25 3 0.045 3 0.18 0.74 重度疲劳
Tab.8 Different fatigue state identification results of NTHU dataset
编号 真实状态 眼部 嘴部 Q 识别结果
BF ECN ECR YF MOR/ MORM
1 清醒 25 1 0.042 0 0 0.27 清醒
1 疲劳 29 2 0.040 1 0.07 0.59 疲劳
1 重度疲劳 38 2 0.053 2 0.11 0.86 重度疲劳
2 清醒 28 0 0.040 0 0 0.23 清醒
2 疲劳 30 0 0.046 1 0.05 0.39 疲劳
2 重度疲劳 35 5 0.050 3 0.2 0.92 重度疲劳
Tab.9 Recognition results of simulating driving fatigue state
[1]   HOODA R, JOSHI V, SHAH M A comprehensive review of approaches to detect fatigue using machine learning techniques[J]. Chronic Diseases and Translational Medicine, 2022, 8 (1): 26- 35
doi: 10.1016/j.cdtm.2021.07.002
[2]   KUWAHARA A, NISHIKAWA K, HIRAKAWA R, et al Eye fatigue estimation using blink detection based on eye aspect ratio mapping(EARM)[J]. Cognitive Robotics, 2022, 2 (1): 50- 59
[3]   ZHANG K, ZHANG Z, LI Z, et al Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23 (10): 1499- 1503
doi: 10.1109/LSP.2016.2603342
[4]   GUO X, LI S, YU J, et al. PFLD: a practical facial landmark detector [EB/OL]. (2019-02-28)[2022-07-07]. https://arxiv.org/abs/ 1902.10859v1.
[5]   ZHANG Z, PING L, CHEN C L, et al. Facial landmark detection by deep multi-task learning [C]// European Conference on Computer Vision. Cham: Springer, 2014: 94-108.
[6]   CHEN L, XIN G, LIU Y, et al Driver fatigue detection based on facial key points and LSTM[J]. Security and Communication Networks, 2021, 2021 (8): 1- 9
doi: 10.1016/S1353-4858(21)00081-7
[7]   WANG J, YU X, LIU Q, et al Research on key technologies of intelligent transportation based on image recognition and anti-fatigue driving[J]. EURASIP Journal on Image and Video Processing, 2019, 2019 (1): 1- 13
doi: 10.1186/s13640-018-0395-2
[8]   ADHINATA F D, RAKHMADANI P, WIJAYANTO D Fatigue detection on face image using FaceNet algorithm and K-nearest neighbor classifier[J]. Journal of Information Systems Engineering and Business Intelligence, 2021, 7 (1): 22- 30
doi: 10.20473/jisebi.7.1.22-30
[9]   PHAM L, VU T H, TRAN T A. Facial expression recognition using residual masking network [C]// 25th International Conference on Pattern Recognition. Milan: IEEE, 2021: 4513-4519.
[10]   LIU M Z, XU X, HU J, et al Real time detection of driver fatigue based on CNN-LSTM[J]. IET Image Processing, 2022, 16 (2): 576- 595
doi: 10.1049/ipr2.12373
[11]   QUDDUS A, ZANDI A S, PREST L, et al Using long short term memory and convolutional neural networks for driver drowsiness detection[J]. Accident Analysis and Prevention, 2021, 156 (1): 106- 107
[12]   WEI S, ZHANG X, WEI Z, et al Driver fatigue driving detection based on eye state[J]. International Journal of Digital Content Technology and its Applications, 2011, 5 (10): 307- 314
doi: 10.4156/jdcta.vol5.issue10.36
[13]   NOMAN T B, AHAD A R Mobile-based eye-blink detection performance analysis on android platform[J]. Frontiers in ICT, 2018, 5 (4): 91- 100
[14]   WORAH G, KHAN A, KOTHARI M, et al Monitor eye-care system using blink detection: a convolutional neural net approach[J]. International Journal of Engineering and Technical Research, 2017, 14 (6): 80- 87
[15]   ZHANG F, SU J, GENG L, et al. Driver fatigue detection based on eye state recognition [C]// 2017 International Conference on Machine Vision and Information Technology. Singapore: IEEE, 2017: 105-110.
[16]   CHINSATIT W, SAITOH T CNN-based pupil center detection for wearable gaze estimation system[J]. Applied Computational Intelligence and Soft Computing, 2017, 11 (5): 1- 10
[17]   YU J, ZHANG Y. A driving early warning method based on multi-factor fusion [C]// 2021 3rd International Conference on Applied Machine Learning. Stockholm: IEEE, 2021: 319-324.
[18]   HU X, HUANG B. Face detection based on SSD and CamShift [C]// 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference. Chongqing: IEEE, 2020: 2324-2328.
[19]   XIANG Y, YANG H, HU R, et al. Comparison of the deep learning methods applied on human eye detection [C]// 2021 IEEE International Conference on Power Electronics, Computer Applications. Shenyang: IEEE, 2021: 314-318.
[20]   YUAN Y, DAI F, SONG Y, et al. On fatigue driving detection system based on deep learning [C]// Chinese Intelligent Systems Conference. Singapore: Springer, 2020: 734-741.
[21]   SUN X, WU P, HOI S Face detection using deep learning: an improved faster RCNN approach[J]. Neurocomputing, 2018, 299 (19): 42- 50
[22]   YAO Z, SONG X, ZHAO L, et al Real-time method for traffic sign detection and recognition based on YOLOv3-tiny with multiscale feature extraction[J]. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 2021, 235 (7): 1978- 1991
doi: 10.1177/0954407020980559
[23]   LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector [C]// European Conference on Computer Vision. Cham: Springer, 2016: 21-37.
[24]   董红召, 方浩杰, 张楠 旋转框定位的多尺度再生物品目标检测算法[J]. 浙江大学学报: 工学版, 2022, 56 (1): 16- 25
DONG Hong-zhao, FANG Hao-jie, ZHANG Nan Multi-scale object detection algorithm for recycled objects based on rotating block positioning[J]. Journal of Zhejiang University: Engineering Science, 2022, 56 (1): 16- 25
[25]   胡峰松, 程哲坤, 徐青云, 等 基于多特征融合的疲劳驾驶状态识别方法研究[J]. 湖南大学学报: 自然科学版, 2022, 49 (4): 100- 107
HU Feng-song, CHENG Zhe-kun, XU Qing-yun, et al Research on fatigue driving state recognition method based on multi-feature fusion[J]. Journal of Hunan University: Natural Sciences, 2022, 49 (4): 100- 107
[26]   张姝玮, 郭忠印, 杨轸, 等 驾驶行为多重分形特征在驾驶疲劳检测中的应用[J]. 吉林大学学报: 工学版, 2021, 51 (2): 557- 564
ZHANG Shu-wei, GUO Zhong-yin, YANG Zhen, et al Application of multi-fractal features of driving performance in driver fatigue detection[J]. Journal of Jilin University: Engineering and Technology Edition, 2021, 51 (2): 557- 564
[1] Qing-lu MA,Jia-ping LU,Xiao-yao TANG,Xue-feng DUAN. Improved YOLOv5s flame and smoke detection method in road tunnels[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(4): 784-794.
[2] Yao ZENG,Fa-qin GAO. Surface defect detection algorithm of electronic components based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(3): 455-465.
[3] Tian-le YUAN,Ju-long YUAN,Yong-jian ZHU,Han-chen ZHENG. Surface defect detection algorithm of thrust ball bearing based on improved YOLOv5[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(12): 2349-2357.
[4] Yun-zuo ZHANG,Wei GUO,Zhao-quan CAI,Wen-bo LI. Remote sensing image target detection combining multi-scale and attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2215-2223.
[5] Hong-zhao DONG,Hao-jie FANG,Nan ZHANG. Multi-scale object detection algorithm for recycled objects based on rotating block positioning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 16-25.