Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (1): 26-35    DOI: 10.3785/j.issn.1008-973X.2022.01.003
    
Recognition method of submarine operation posture based on convolutional pose machine
Jing-luan WANG(),Deng-kai CHEN*(),Meng-ya ZHU,Han-yu WANG,Yi-wei SUN
Shaanxi Engineering Laboratory for Industrial Design, Northwestern Polytechnical University, Xi'an 710072, China
Download: HTML     PDF(1382KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A new posture recognition and analysis method based on convolutional pose machines was proposed aiming at the problems of complicated recognition process and low recognition accuracy in the existing submariner’s operation posture recognition and analysis methods. The human body posture features were structured and coded, and the spatial and projected coordinate system were constructed to explain the human body posture. The calculation formulae of limb angle and the judging processes of special limb state were defined. The spatial and texture features of the RGB operation posture image can be extracted by building the submariner’s operation posture recognition algorithm. The joint points, limb angles and state data of the submariner’s operation posture can be output. The application of the proposed method was verified by the submariner’s operation posture sample data set constructed by collecting submarine operation posture image. The percentage of correct keypoints index value of the recognition algorithm reached 81.2% in the algorithm test. The average accuracy rate of the algorithm in identifying the joint points reached 87.7% in the application verification experiment. The experimental results show that the method is reliable in the recognition and analysis of the submariner’s operation posture, and can effectively identify and analyze the negative factors of the submariner’s operation posture.

Key words： industrial design      posture recognition      submariner’s operation posture      convolutional pose machine      posture analysis     
Received: 12 May 2021      Published: 05 January 2022
CLC:  TB 472  
Fund:  中央高校基本科研业务费资助项目(31020190504007)；陕西省特支计划领军人才资助项目(w099115)
Corresponding Authors: Deng-kai CHEN     E-mail: wangjingluan1@163.com;chendengkai@nwpu.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Jing-luan WANG
Deng-kai CHEN
Meng-ya ZHU
Han-yu WANG
Yi-wei SUN
Cite this article:

Jing-luan WANG,Deng-kai CHEN,Meng-ya ZHU,Han-yu WANG,Yi-wei SUN. Recognition method of submarine operation posture based on convolutional pose machine. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 26-35.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.01.003     OR     https://www.zjujournals.com/eng/Y2022/V56/I1/26


基于卷积姿态机的潜航员作业姿态识别方法

针对现有潜航员作业姿态识别分析方法中识别过程繁琐、识别精度低的问题，提出基于卷积姿态机的潜航员作业姿态识别分析方法. 对人体姿态特征进行结构化编码，构建空间及投影坐标系进行解析，定义肢体角度计算公式与肢体特殊状态判断流程. 通过搭建潜航员作业姿态识别算法，实现作业姿态RGB图像空间特征与纹理特征的提取，输出潜航员作业姿态关节点、肢体角度与状态数据. 通过采集潜航员作业姿态图像构建潜航员作业姿态样本数据集，对所提方法进行应用验证. 在算法测试中，识别算法的PCK指标值达到81.2%. 在应用验证实验中，算法识别关节点的平均准确率达到87.7%. 该方法在潜航员作业姿态识别分析上是可靠的，可以有效地识别与分析潜航员作业姿态中的危险因素.


关键词： 工业设计,  姿态识别,  潜航员作业姿态,  卷积姿态机,  姿态分析 
编码 含义 编码 含义
1 头部 9 左手腕
2 颈部 10 左手指
3 右肩 11 右髋
4 右手肘 12 右膝
5 右手腕 13 右踝
6 右手指 14 左髋
7 左肩 15 左膝
8 左手肘 16 左踝
Tab.1 Human body posture joint point coding
Fig.1 Spatial and sagittal projection coordinate system definition
编码 含义 编码 含义
1-2 颈部 2-9-12 腰部
2-3-4 右大臂 3-4-5 右小臂
4-5-6 右手腕 2-7-8 左大臂
7-8-9 左小臂 8-9-10 左手腕
11-12-13 右腿 14-15-16 左腿
Tab.2 Human body limb coding
Fig.2 Comparison of trunk normal state and twisted state
Fig.3 Schematic diagram of shoulders raised
Fig.4 Principles of submariner's operation posture recognition algorithm
Fig.5 Structure of submariner’s operation posture recognition algorithm
序号 输入 运算 参数 输出
1 原始图像 — — 368×368×3
2 368×368×3 Conv.+ReLU k = 9, n = 128, s = 1 368×368×128
3 368×368×128 max_pool 2×2, s = 2 184×184×128
4 184×184×128 Conv.+ReLU k = 9, n = 128, s = 1 184×184×128
5 184×184×128 max_pool 2×2, s =2 92×92×128
6 92×92×128 Conv.+ReLU k = 9, n = 128, s = 1 92×92×128
7 92×92×128 max_pool 2×2, s = 2 46×46×128
8 46×46×128 Conv.+ReLU k = 5, n = 32, s = 1 46×46×32
9 46×46×32 Conv.+ReLU k = 9, n = 512, s = 1 46×46×512
10 46×46×512 Conv.+ReLU k = 1, n = 512, s = 1 46×46×512
11 46×46×512 Conv.+ReLU k = 1, n = 15, s = 1 46×46×15
Tab.3 Parameters of algorithm g1-1 convolution group
序号 输入 运算 参数 输出
1 原始图像 — — 368×368×3
2 368×368×3 Conv.+ReLU k = 9, n = 128, s = 1 368×368×128
3 368×368×128 max_pool 2×2, s = 2 184×184×128
4 184×184×128 Conv.+ReLU k = 9, n = 128, s = 1 184×184×128
5 184×184×128 max_pool 2×2, s = 2 92×92×128
6 92×92×128 Conv.+ReLU k = 9, n = 128, s = 1 92×92×128
7 92×92×128 max_pool 2×2, s = 2 46×46×128
8 46×46×128 Conv.+ReLU k = 5, n = 32, s = 1 46×46×32
9 46×46×32+
46×46×15+
46×46×1 		Concat ( ) — 46×46×48
10 46×46×48 Conv.+ReLU k = 11, n = 128, s = 1 46×46×128
11 46×46×128 Conv.+ReLU k = 11, n = 128, s = 1 46×46×128
12 46×46×128 Conv.+ReLU k = 11, n = 128, s = 1 46×46×128
13 46×46×128 Conv.+ReLU k = 1, n = 128, s = 1 46×46×128
14 46×46×128 Conv.+ReLU k = 1, n = 15, s = 1 46×46×15
Tab.4 Parameters of algorithm g2-1 andg2-2 convolution group
Fig.6 Training process of submariner's operation posture recognition algorithm
Fig.7 Characteristic diagram of human joint points in six stages
方法 头部 颈部 肩部 肘部 腕部 臀部 膝部 踝部 平均
FC[19] 75.8 71.3 67.3 64.3 60.4 63.7 60.1 55.7 64.8
DC[20] 83.6 74.6 73.9 71.1 66.0 70.9 64.1 59.4 70.5
CPN[21] 85.4 77.1 79.6 74.5 69.9 73.6 70.9 64.8 74.5
PAF[13] 82.6 75.8 75.9 71.3 68.5 68.9 65.9 60.9 71.2
文献[17] 91.8 82.1 80.6 70.9 82.1 81.9 74.6 80.2 80.5
文献[18] 90.6 84.3 84.5 75.0 78.0 83.8 78.9 79.2 81.8
本文方法 93.6 83.2 90.5 83.2 73.6 77.9 76.2 71.7 81.2
Tab.5 Comparison of PCK indicators of different algorithms %
Fig.8 Manned submersible simulation operation interface layout
Fig.9 Human skeleton generated by submariner’s operation posture recognition algorithm
Fig.10 Score of 16 joint points recognition accuracy
作业姿态 $\bar \theta (\sigma ) $/(°) ICC
颈部 腰部 右大臂 右小臂 右手腕 左大臂 左小臂 左手腕 右腿 左腿
1 1.2(1.3) 2.2(2.9) 14.5(10.7) 72.5(9.8) 1.8(1.2) 14.8(12.3) 70.0(5.8) 1.7(1.7) 105(8.0) 107.2(11.5) 0.966
2 8.6(6.8) 6.9(1.7) 29.4(9.7) 42.2(14.0) 6.3(4.5) 19.4(10.0) 65.0(4.3) 1.7(1.5) 103.1(10.1) 105.4(10.2) 0.951
3 ?6.7(3.9) 8.2(1.9) 39.3(7.4) 30.7(13.1) 7.7(5.9) 22.6(9.2) 67.7(10.2) 1.9(1.5) 104.4(10.5) 105.2(11.6) 0.953
4 ?17.9(10.0) 28.2(3.7) ?9.9(33.2) 74.4(32.0) 2.2(2.0) 77.7(11.2) 18.5(6.8) 8.4(4.5) 101.8(10.8) 104.6(10.3) 0.879
5 ?19.7(8.1) 35.9(6.1) ?11.4(4.4) 21.5(11.4) 16.6(4.5) 82.8(8.9) 18.1(9.0) 12.5(4.5) 67.3(21.9) 68.9(24.9) 0.888
6 7.7(1.5) 5.6(0.2) 21.2(7.6) 40.5(19.3) 22.2(2.0) 23.6(7.1) 34.7(10.5) 22.0(0.5) 108.2(12.7) 106.4(10.5) 0.944
7 20.4(10.0) 30.5(14.0) ?19.6(10.5) 119.5(22.9) 2.2(1.2) ?14.1(13.4) 112.6(18.3) 2.0(1.8) 106.1(9.2) 109.3(11.3) 0.950
8 ?4.9(9.6) 19.5(13.9) ?10.5(9.5) 91.4(9.7) 2.0(1.0) ?5.0(8.1) 87.6(9.4) 1.9(1.6) 104.8(2.4) 107.1(4.7) 0.971
9 ?16.9(9.9) 16.0(10.9) 8.1(9.4) 82.6(12.2) 3.3(2.0) 10.3(7.5) 82.1(8.6) 1.4(0.7) 104.2(1.0) 104.5(3.0) 0.976
10 ?6.3(3.2) 16.6(8.4) 17.0(17.7) 61.7(27.6) 9.5(3.1) 8.9(2.5) 76.9(11.9) 1.8(1.1) 108.4(4.9) 107.6(7.2) 0.921
11 ?17.4(3.9) 15.9(7.1) 16.8(15.4) 65.1(24.2) 10.7(5.7) 8.8(4.9) 71.6(15.6) 1.9(1.6) 106.2(5.4) 107.1(3.5) 0.933
12 ?14.8(5.3) 26.2(5.6) ?10.0(6.8) 82.8(15.2) 1.7(1.7) 75.6(11.1) 20.0(6.5) 9.1(3.7) 108.9(11.7) 109.4(12.7) 0.963
13 ?23.4(6.3) 24.1(6.8) ?10.4(4.4) 90.9(13.3) 1.9(1.8) 78.4(10.8) 16.8(8.2) 8.3(3.4) 103.7(10.2) 107.4(12.3) 0.965
14 ?14.0(11.1) 39.2(7.7) 10.7(10.2) 19(6.6) 16.7(5.3) 79.8(13.6) 23.1(14.3) 12.7(5.4) 65.0(26.0) 67.7(28.7) 0.832
15 ?24.9(9.0) 37.3(6.3) 10.3(4.6) 21(5.3) 16.1(4.7) 81.2(10.8) 21.2(11.1) 12.2(4.7) 68.0(27.9) 69.8(27.8) 0.874
ICC 0.935 0.923 0.976 0.974 0.853 0.881 0.956 0.887 0.852 0.819 —
Tab.6 Average value (standard deviation) of limb angle and ICC result of submariner’s operation posture recognition
Fig.11 Limb angle change data of operation video
[1]   CHEN D K, FAN Y, LI W H, et al Human reliability prediction in deep-sea sampling process of the manned submersible[J]. Safety Science, 2019, 112 (2): 1- 8
[2]   张帅, 何卫平, 陈登凯, 等 载人潜水器舱室空间舒适性复合评估方法[J]. 哈尔滨工业大学学报, 2019, 51 (10): 83- 89
ZHANG Shuai, HE Wei-ping, CHEN Deng-kai, et al Compound evaluation method for the space comfort of manned submersible[J]. Journal of Harbin Institute of Technology, 2019, 51 (10): 83- 89
doi: 10.11918/j.issn.0367-6234.201810148
[3]   张宁, 李亚军, 段齐骏, 等 面向老年俯身作业的人机工程舒适性设计[J]. 浙江大学学报: 工学版, 2017, 51 (1): 95- 105
ZHANG Ning, LI Ya-jun, DUAN Qi-jun, et al Man-machine comfortable designing for bending operations of elderly[J]. Journal of Zhejiang University: Engineering Science, 2017, 51 (1): 95- 105
[4]   TRASK C, MATHIASSEN S E, ROSTAMI M, et al Observer variability in posture assessment from video recordings: The effect of partly visible periods[J]. Applied Ergonomics, 2017, 60 (4): 275- 281
[5]   朱德慰, 李志海, 吴镇炜. 基于异常行为监测的人机安全协作方法[EB/OL]. [2021-04-21]. http://kns.cnki.net/kcms/detail/11.5946.TP.20210329.1645.018.html.
ZHU De-wei, LI Zhi-hai, WU Zhen-wei. Abnormal behavior monitoring based method for safe human-robot collaboration [EB/OL]. [2021-04-21]. http://kns.cnki.net/kcms/detail/11.5946.TP.20210329.1645.018.html.
[6]   MANGHISI V M, UVA A E, FIORENTINO M, et al Real time RULA assessment using Kinect v2 sensor[J]. Applied Ergonomics, 2017, 65 (11): 481- 491
[7]   PLANTARD P, SHUM H P, PIERRES A S, et al Validation of an ergonomic assessment method using kinect data in real workplace conditions[J]. Applied Ergonomics, 2017, 65 (11): 562- 569
[8]   TOPLEY M, RICHARDS J G A comparison of currently available optoelectronic motion capture systems[J]. Journal of Biomechanics, 2020, 106 (6): 109820
[9]   YAN X Z, LI H, ANGUS R L, et al Wearable IMU-based real-time motion warning system for construction workers' musculoskeletal disorders prevention[J]. Automation in Construction, 2017, 74 (2): 2- 11
[10]   HUYNH-THE T, HUA C H, NGO T T, et al Image representation of pose-transition feature for 3D skeleton-based action recognition[J]. Information Sciences, 2020, 513 (5): 112- 126
[11]   袁公萍, 汤一平, 韩旺明, 等 基于深度卷积神经网络的车型识别方法[J]. 浙江大学学报: 工学版, 2018, 52 (4): 694- 702
YUAN Gong-ping, TANG Yi-ping, HAN Wang-ming, et al Vehicle category recognition based on deep convolutional neural network[J]. Journal of Zhejiang University: Engineering Science, 2018, 52 (4): 694- 702
[12]   LECUN Y, BENGIO Y, HINTON G Deep learning[J]. Nature, 2015, 521 (3): 436- 444
[13]   CAO Z, SIMOM T, WEI S E, et al. Realtime multi-person 2d pose estimation using part affinity fields [C]// Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake: IEEE, 2018: 1302-1310.
[14]   OBERWEGER M, WOHLHART P, LEPETIT V Hands deep in deep learning for hand pose estimation[J]. Computer Sciences, 2015, 24 (12): 21- 30
[15]   WEI S E, RAMAKRISHNA V, KANADE T, et al. Convolutional pose machines [C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 4724-4732.
[16]   PROMSRI A, HAID T, FEDEROLF P How does lower limb dominance influence postural control movements during single leg stance?[J]. Human Movement Science, 2018, 58 (4): 165- 174
[17]   PFISTER T, CHARLES J, ZISSERMAN A. Flowing ConvNets for human pose estimation in videos [C]// Proceedings of the 2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1913-1921.
[18]   PISHCHULIN L, INSAFUTDINOV E, TANG S, et al. DeepCut: joint subset partition and labeling for multi person pose estimation [C]// Proceeding of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 4929-4937.
[19]   CHEN Y, WANG Z C, PENG Y X, et al. Cascaded pyramid network for multi-person pose estimation [C]// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake: IEEE, 2018: 7103-7112.
[20]   SU K, YU D D, XU Z Q, et al. Multi-person pose estimation with enhanced channel-wise and spatial information [C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5674-5682.
[21]   ZHANG J, CHEN Z, TAO D. Towards high performance human keypoint detection [C]// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020.
[1] Yan-pu YANG,Zheng GONG,Chen-xin LAN,Zi-jing LEI,Xin-rui WANG. Construction of industrial design decision-making network and its dynamic evolution simulation[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(12): 2298-2306.
[2] Jian CHEN,Rong MO,Sui-huai YU,Jian-jie CHU,Deng-kai CHEN,Jing GONG. Multi-objective group decision method for crowdsourcing product modeling design scheme in cloud environment[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(8): 1517-1524.
[3] CHUANG Yue-long, CHEN Ling, CHEN Gen-cai. Hierarchical Bag-of-Features with Spectral-HIK filter based hand posture recognition[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(9): 1531-1536.
[4] SHU Shang-Shang, LUO Shi-Jian, YING Fang-Tian, HE Ji. Product family design DNA for product visual identity[J]. Journal of ZheJiang University (Engineering Science), 2010, 44(4): 715-721.
[5] LIU Zheng, SUN Shou-qian, PAN Yun-he. Information framework based designers' cognitive strategies classification and its application[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(5): 884-889.