Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2023, Vol. 57 Issue (7): 1287-1296    DOI: 10.3785/j.issn.1008-973X.2023.07.003
自动化技术     
多特征融合的驾驶员疲劳状态检测方法
方浩杰1(),董红召1,*(),林少轩1,罗建宇2,方勇2
1. 浙江工业大学 智能交通系统联合研究所，浙江 杭州 310014
2. 杭州金通科技集团股份有限公司，浙江 杭州 310014
Driver fatigue state detection method based on multi-feature fusion
Hao-jie FANG1(),Hong-zhao DONG1,*(),Shao-xuan LIN1,Jian-yu LUO2,Yong FANG2
1. ITS Joint Research Institute, Zhejiang University of Technology, Hangzhou 310014, China
2. Hangzhou Jintong Technology Group Limited Company, Hangzhou 310014, China
 全文: PDF(1481 KB)   HTML
摘要：

针对现有疲劳状态检测方法无法适用于疫情防控下的驾驶员，利用改进后的YOLOv5目标检测算法，对驾驶员的面部区域进行检测，建立多特征融合的疲劳状态检测方法. 针对公交驾驶特性，建立包含佩戴口罩和未佩戴口罩情况的图像标签数据. 通过增加YOLOv5模型的特征采样次数，提高眼、嘴、面部区域的检测精度. 利用BiFPN网络结构保留多尺度的特征信息，使得预测网络对不同大小的目标更敏感，提升整体模型的检测能力. 结合人脸关键点算法提出参数补偿机制，提高眨眼、打哈欠帧数的准确率. 将多种疲劳参数融合归一化处理，开展疲劳等级划分. 公开数据集NTHU和自制数据集的验证结果表明，该方法对佩戴口罩和未佩戴口罩情况均可以进行眨眼、打哈欠识别，可以准确地判断驾驶员的疲劳状态.
关键词： 驾驶安全;  疲劳检测;  YOLOv5;  视频分析;  模拟驾驶    
Abstract:

The improved YOLOv5 object detection algorithm was used to detect the facial region of the driver and a multi-feature fusion fatigue state detection method was established aiming at the problem that existing fatigue state detection method cannot be applied to drivers under the epidemic prevention and control. The image tag data including the situation of wearing a mask and the situation without wearing a mask were established according to the characteristics of bus driving. The detection accuracy of eyes, mouth and face regions was improved by increasing the feature sampling times of YOLOv5 model. The BiFPN network structure was used to retain multi-scale feature information, which makes the prediction network more sensitive to targets of different sizes and improves the detection ability of the overall model. A parameter compensation mechanism was proposed combined with face keypoint algorithm in order to improve the accuracy of blink and yawn frame number. A variety of fatigue parameters were fused and normalized to conduct fatigue classification. The results of the public dataset NTHU and the self-made dataset show that the proposed method can recognize the blink and yawn of drivers both with and without masks, and can accurately judge the fatigue state of drivers.
Key words: driver safety    fatigue detection    YOLOv5    video analytics    driving simulation
收稿日期: 2022-08-06 出版日期: 2023-07-17
CLC:  TP 391  
基金资助: 国家自然科学基金资助项目（61773347）；浙江公益技术研究资助项目（LGF19F030001）
通讯作者: 董红召     E-mail: 1047677234@qq.com;its@zjut.edu.cn
作者简介: 方浩杰（1997—），男，硕士生，从事智能交通的研究. orcid.org/0000-0003-1841-0529. E-mail： 1047677234@qq.com
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
方浩杰
董红召
林少轩
罗建宇
方勇

引用本文:

方浩杰,董红召,林少轩,罗建宇,方勇. 多特征融合的驾驶员疲劳状态检测方法[J]. 浙江大学学报(工学版), 2023, 57(7): 1287-1296.

Hao-jie FANG,Hong-zhao DONG,Shao-xuan LIN,Jian-yu LUO,Yong FANG. Driver fatigue state detection method based on multi-feature fusion. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1287-1296.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.07.003    或    https://www.zjujournals.com/eng/CN/Y2023/V57/I7/1287

图 1  改进后的YOLOv5网络结构
图 2  图像标注的示意图
图 3  人脸关键点位置的示意图
图 4  NTHU数据集的部分视频图像
图 5  自制数据集图像的示意图
模型 M/106 FLOPs/109 AP mAP/%
Um Ym Ce Oe Cm Om Ya
Faster-RCNN [21] 137.1 370.4 99.8 100 73.1 74.2 97.2 98.2 90.5 90.6
YOLOv3-tiny [22] 8.7 13.0 99.0 99.0 90.3 91.0 94.1 94.3 94.0 94.5
SSD[23] 26.3 62.8 99.5 99.8 84.2 90.0 89.8 95.4 98.0 93.8
YOLOv5-S[24] 7.0 15.9 99.0 99.3 91.8 92.5 94.5 95.0 94.5 95.2
YOLOv5-S+Detection layer 8.2 28.3 98.7 99.1 94.8 95.7 97.3 98.8 97.6 97.5
YOLOv5-S+BiFPN 7.1 16.6 99.2 99.4 93.2 94.4 96.3 97.3 98.0 96.8
YOLOv5-S+ Detection layer+BiFPN (本文模型) 9.3 29.9 99.5 99.5 98.6 99.5 99.5 99.5 99.5 99.4
表 1  不同模型的性能比较
图 6  本文改进后模型的检测效果图
$ {N_{\rm{e}}} $ $ {N_{{\rm{Pe}}}} $ $ {N_{{\rm{Te}}}} $ $ {R_{\rm{e}}} $
100 436 325 1.34
500 2612 1988 1.31
1000 5223 3962 1.32
表 2  闭眼帧数的统计
$ {N_{\rm{m}}} $ $ {N_{{\rm{Pm}}}} $ $ {N_{{\rm{Tm}}}} $ $ {R_{\rm{m}}} $
20 1832 1512 1.21
40 3756 3109 1.21
100 9521 7826 1.22
表 3  打哈欠帧数统计
人员编号 戴口罩 $ {N_{\rm{e}}} $ $ {N_{{\rm{De}}}} $ $ {N_{{\rm{Ae}}}} $ $ {N_{{\rm{Fe}}}} $ $ {N_{{\rm{Le}}}} $ $ {A_{\rm{e}}} $/%
1 是 512 515 507 8 5 97.5
2 是 474 480 474 6 0 98.8
3 是 624 647 620 27 4 95.2
4 否 681 690 671 19 10 95.8
5 否 563 572 558 14 5 96.7
6 否 415 426 413 13 2 96.5
表 4  眨眼计数分析
人员编号 戴口罩 $ {N_{\rm{m}}} $ $ {N_{{\rm{Dm}}}} $ $ {N_{{\rm{Am}}}} $ $ {N_{{\rm{Fm}}}} $ $ {N_{{\rm{Lm}}}} $ $ {A_{\rm{m}}} $/%
1 是 6 6 6 0 0 100
2 是 1 1 1 0 0 100
3 是 7 8 7 1 0 87.5
4 否 11 10 10 0 1 90.1
5 否 5 5 5 0 0 100
6 否 3 3 3 0 0 100
表 5  哈欠计数分析
疲劳参数 $ {P_{\min }} $ $ {P_{\max }} $ W
BF 13 34 0.1
ECN 0 5 0.2
ECR 0.02 0.06 0.3
YF 0 3 0.2
MOR 0 0.2 0.2
表 6  综合疲劳参数的最大值以及权重
疲劳状态 Q
清醒 Q < 0.3
疲劳 0.3 ≤ Q < 0.7
重度疲劳 Q ≥ 0.7
表 7  综合疲劳指标与疲劳等级的对应关系
组数 真实状态 眼部 嘴部 Q 识别结果
BF ECN ECR YF MOR
1 清醒 23 0 0.022 1 0.05 0.18 清醒
1 疲劳 25 3 0.053 2 0.1 0.65 疲劳
1 重度疲劳 34 4 0.050 2 0.12 0.74 重度疲劳
2 清醒 25 1 0.035 0 0 0.21 清醒
2 疲劳 20 2 0.047 3 0.15 0.62 疲劳
2 重度疲劳 20 5 0.055 3 0.2 0.89 重度疲劳
3 清醒 22 0 0.042 0 0 0.21 清醒
3 疲劳 17 1 0.039 3 0.17 0.57 疲劳
3 重度疲劳 25 3 0.045 3 0.18 0.74 重度疲劳
表 8  NTHU数据集的不同疲劳状态识别结果
编号 真实状态 眼部 嘴部 Q 识别结果
BF ECN ECR YF MOR/ MORM
1 清醒 25 1 0.042 0 0 0.27 清醒
1 疲劳 29 2 0.040 1 0.07 0.59 疲劳
1 重度疲劳 38 2 0.053 2 0.11 0.86 重度疲劳
2 清醒 28 0 0.040 0 0 0.23 清醒
2 疲劳 30 0 0.046 1 0.05 0.39 疲劳
2 重度疲劳 35 5 0.050 3 0.2 0.92 重度疲劳
表 9  模拟驾驶疲劳状态的识别结果
1 HOODA R, JOSHI V, SHAH M A comprehensive review of approaches to detect fatigue using machine learning techniques[J]. Chronic Diseases and Translational Medicine, 2022, 8 (1): 26- 35
doi: 10.1016/j.cdtm.2021.07.002
2 KUWAHARA A, NISHIKAWA K, HIRAKAWA R, et al Eye fatigue estimation using blink detection based on eye aspect ratio mapping(EARM)[J]. Cognitive Robotics, 2022, 2 (1): 50- 59
3 ZHANG K, ZHANG Z, LI Z, et al Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23 (10): 1499- 1503
doi: 10.1109/LSP.2016.2603342
4 GUO X, LI S, YU J, et al. PFLD: a practical facial landmark detector [EB/OL]. (2019-02-28)[2022-07-07]. https://arxiv.org/abs/ 1902.10859v1.
5 ZHANG Z, PING L, CHEN C L, et al. Facial landmark detection by deep multi-task learning [C]// European Conference on Computer Vision. Cham: Springer, 2014: 94-108.
6 CHEN L, XIN G, LIU Y, et al Driver fatigue detection based on facial key points and LSTM[J]. Security and Communication Networks, 2021, 2021 (8): 1- 9
doi: 10.1016/S1353-4858(21)00081-7
7 WANG J, YU X, LIU Q, et al Research on key technologies of intelligent transportation based on image recognition and anti-fatigue driving[J]. EURASIP Journal on Image and Video Processing, 2019, 2019 (1): 1- 13
doi: 10.1186/s13640-018-0395-2
8 ADHINATA F D, RAKHMADANI P, WIJAYANTO D Fatigue detection on face image using FaceNet algorithm and K-nearest neighbor classifier[J]. Journal of Information Systems Engineering and Business Intelligence, 2021, 7 (1): 22- 30
doi: 10.20473/jisebi.7.1.22-30
9 PHAM L, VU T H, TRAN T A. Facial expression recognition using residual masking network [C]// 25th International Conference on Pattern Recognition. Milan: IEEE, 2021: 4513-4519.
10 LIU M Z, XU X, HU J, et al Real time detection of driver fatigue based on CNN-LSTM[J]. IET Image Processing, 2022, 16 (2): 576- 595
doi: 10.1049/ipr2.12373
11 QUDDUS A, ZANDI A S, PREST L, et al Using long short term memory and convolutional neural networks for driver drowsiness detection[J]. Accident Analysis and Prevention, 2021, 156 (1): 106- 107
12 WEI S, ZHANG X, WEI Z, et al Driver fatigue driving detection based on eye state[J]. International Journal of Digital Content Technology and its Applications, 2011, 5 (10): 307- 314
doi: 10.4156/jdcta.vol5.issue10.36
13 NOMAN T B, AHAD A R Mobile-based eye-blink detection performance analysis on android platform[J]. Frontiers in ICT, 2018, 5 (4): 91- 100
14 WORAH G, KHAN A, KOTHARI M, et al Monitor eye-care system using blink detection: a convolutional neural net approach[J]. International Journal of Engineering and Technical Research, 2017, 14 (6): 80- 87
15 ZHANG F, SU J, GENG L, et al. Driver fatigue detection based on eye state recognition [C]// 2017 International Conference on Machine Vision and Information Technology. Singapore: IEEE, 2017: 105-110.
16 CHINSATIT W, SAITOH T CNN-based pupil center detection for wearable gaze estimation system[J]. Applied Computational Intelligence and Soft Computing, 2017, 11 (5): 1- 10
17 YU J, ZHANG Y. A driving early warning method based on multi-factor fusion [C]// 2021 3rd International Conference on Applied Machine Learning. Stockholm: IEEE, 2021: 319-324.
18 HU X, HUANG B. Face detection based on SSD and CamShift [C]// 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference. Chongqing: IEEE, 2020: 2324-2328.
19 XIANG Y, YANG H, HU R, et al. Comparison of the deep learning methods applied on human eye detection [C]// 2021 IEEE International Conference on Power Electronics, Computer Applications. Shenyang: IEEE, 2021: 314-318.
20 YUAN Y, DAI F, SONG Y, et al. On fatigue driving detection system based on deep learning [C]// Chinese Intelligent Systems Conference. Singapore: Springer, 2020: 734-741.
21 SUN X, WU P, HOI S Face detection using deep learning: an improved faster RCNN approach[J]. Neurocomputing, 2018, 299 (19): 42- 50
22 YAO Z, SONG X, ZHAO L, et al Real-time method for traffic sign detection and recognition based on YOLOv3-tiny with multiscale feature extraction[J]. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 2021, 235 (7): 1978- 1991
doi: 10.1177/0954407020980559
23 LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector [C]// European Conference on Computer Vision. Cham: Springer, 2016: 21-37.
24 董红召, 方浩杰, 张楠 旋转框定位的多尺度再生物品目标检测算法[J]. 浙江大学学报: 工学版, 2022, 56 (1): 16- 25
DONG Hong-zhao, FANG Hao-jie, ZHANG Nan Multi-scale object detection algorithm for recycled objects based on rotating block positioning[J]. Journal of Zhejiang University: Engineering Science, 2022, 56 (1): 16- 25
25 胡峰松, 程哲坤, 徐青云, 等 基于多特征融合的疲劳驾驶状态识别方法研究[J]. 湖南大学学报: 自然科学版, 2022, 49 (4): 100- 107
HU Feng-song, CHENG Zhe-kun, XU Qing-yun, et al Research on fatigue driving state recognition method based on multi-feature fusion[J]. Journal of Hunan University: Natural Sciences, 2022, 49 (4): 100- 107
26 张姝玮, 郭忠印, 杨轸, 等 驾驶行为多重分形特征在驾驶疲劳检测中的应用[J]. 吉林大学学报: 工学版, 2021, 51 (2): 557- 564
ZHANG Shu-wei, GUO Zhong-yin, YANG Zhen, et al Application of multi-fractal features of driving performance in driver fatigue detection[J]. Journal of Jilin University: Engineering and Technology Edition, 2021, 51 (2): 557- 564
[1] 马庆禄,鲁佳萍,唐小垚,段学锋. 改进YOLOv5s的公路隧道烟火检测方法[J]. 浙江大学学报(工学版), 2023, 57(4): 784-794.
[2] 曾耀,高法钦. 基于改进YOLOv5的电子元件表面缺陷检测算法[J]. 浙江大学学报(工学版), 2023, 57(3): 455-465.
[3] 袁天乐,袁巨龙,朱勇建,郑翰辰. 基于改进YOLOv5的推力球轴承表面缺陷检测算法[J]. 浙江大学学报(工学版), 2022, 56(12): 2349-2357.
[4] 张云佐,郭威,蔡昭权,李文博. 联合多尺度与注意力机制的遥感图像目标检测[J]. 浙江大学学报(工学版), 2022, 56(11): 2215-2223.
[5] 董红召,方浩杰,张楠. 旋转框定位的多尺度再生物品目标检测算法[J]. 浙江大学学报(工学版), 2022, 56(1): 16-25.
[6] 王龙,余隋怀,初建杰. 基于眼动分析的车内信息显示位置设计[J]. 浙江大学学报(工学版), 2020, 54(4): 671-677.