Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2026, Vol. 60 Issue (10): 2196-2206    DOI: 10.3785/j.issn.1008-973X.2026.10.012
    
Multimodal gait recognition based on pose induction
Wei GAO1(),Zhidong YU1,Yuanyuan CHANG2,Wang MIAO1,Yongliang YIN2,Hongxia DENG1,*()
1. College of Computer Science and Technology (College of Data Science), Taiyuan University of Technology, Taiyuan 030024, China
2. School of Physical Education and Health Engineering, Taiyuan University of Technology, Taiyuan 030024, China
Download: HTML     PDF(1270KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A pose-induced multimodal gait recognition method was proposed and a new multimodal dataset named CASIA-V containing real 3D human poses was constructed, in order to achieve multi-view fine-grained gait recognition in complex environments. The dataset was collected using multiple synchronized devices, capturing 93 subjects walking under three conditions (normal, carrying a bag, and wearing a coat) from 11 different viewpoints. It contained 11 253 RGB sequences along with corresponding motion-capture poses and depth maps. At the methodological level, a spatiotemporal alignment strategy was introduced to match motion-capture poses with video-based poses. A separately trainable 3D Pose-Induced Module (3D-SIM) was designed to extract real pose semantics, and a dual-branch model named SIGait was constructed to fuse pose and silhouette features. The experiment showed that, under the cross-view evaluation protocol of the CASIA-V dataset (excluding samples from the same view), SIGait achieved a Rank-1 accuracy of 97.7% in the normal walking scenario. Compared with the baseline method MSAFF, SIGait improved the Rank-1 accuracy by 5.1 and 7.1 percentage points in the carrying-bag and wearing-coat scenarios, respectively. Notably, the inference process did not require motion-capture data, validating that real 3D pose effectively enhanced gait recognition in complex scenarios. The results further showed that real 3D pose semantics enhanced multimodal feature representation and maintained stable recognition performance under view variations and appearance interference.



Key wordsgait recognition      multi-modal dataset      3D pose      spatiotemporal alignment      pose induction      feature fusion      inertial motion capture     
Received: 16 December 2025      Published: 28 July 2026
CLC:  TP 393  
Fund:  山西省中央引导地方科技发展资金项目(YDZJSX2022A016);山西省重点研发计划资助项目(2022ZDYF128);山西省科技战略项目(202404030401080).
Corresponding Authors: Hongxia DENG     E-mail: gw190428@163.com;denghongxia@tyut.edu.cn
Cite this article:

Wei GAO,Zhidong YU,Yuanyuan CHANG,Wang MIAO,Yongliang YIN,Hongxia DENG. Multimodal gait recognition based on pose induction. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2196-2206.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2026.10.012     OR     https://www.zjujournals.com/eng/Y2026/V60/I10/2196


基于姿态诱导的多模态步态识别

为了实现复杂环境下步态的多视角细粒度识别,提出基于姿态诱导的多模态步态识别方法,并构建包含真实三维姿态的多模态数据集(CASIA-V). 该数据集通过多设备同步采集93名受试者在正常、背包、穿大衣行走下的11视角数据,包含11253个RGB序列及对应动捕姿态与深度图. 在方法层面,提出时空对齐策略实现动捕与视频姿态匹配,设计可独立训练的三维姿态诱导模块(3D-SIM)提取真实姿态语义,并构建双分支模型SIGait融合姿态与轮廓特征. 实验表明,在CASIA-V数据集的跨视角评估协议下(排除相同视角样本对),SIGait在正常行走场景下的Rank-1准确率达97.7%;相较基线方法MSAFF,在背包与穿大衣场景下Rank-1准确率分别提升5.1和7.1个百分点,且推理过程无需动捕数据,验证了真实三维姿态对复杂场景步态识别的有效促进. 结果同时显示,真实三维姿态语义能增强多模态特征表达,使识别系统在视角变化与外观干扰条件下保持稳定识别性能.


关键词: 步态识别,  多模态数据集,  三维姿态,  时空对齐,  姿态诱导,  特征融合,  惯性动捕 
Fig.1 Data collection site diagram
Fig.2 Schematic diagram of time alignment
Fig.3 Schematic diagram of spatial alignment
Fig.4 Examples of different gait data in CASIA-V
Fig.5 Distribution of age, gender and sequence length in CASIA-V
Fig.6 Structure of SIGait network
Fig.7 Structure of 3D-SIM network
池化策略R-1/%
NMBGCL均值
最大池化96.895.189.793.9
全局池化97.295.790.294.4
平均池化97.796.390.994.9
Tab.6 Performance comparison of different pooling strategies
方法R-1/%
NMBGCL均值
GaitGraph[13]84.774.866.375.2
PoseGait[12]69.745.137.250.6
GaitSet[5]92.887.269.983.3
TransGait[19]96.494.485.392.0
SkeletonGait++[21]95.990.984.790.5
MSAFF[18]94.691.283.889.8
SIGait(本研究)97.796.390.994.9
Tab.1 Cross-view comparison results of different methods on CASIA-V dataset
方法R-1/%
NMBGCL均值
GaitGraph[13]84.774.866.375.2
GaitGraph(3D)69.059.653.360.6
GaitGraph(3D)+89.484.075.883.0
PoseGait(3D)[12]68.744.536.049.7
PoseGait(3D)+91.184.977.884.6
MSAFF[18]94.691.283.889.8
MSAFF(3D)90.388.580.686.4
SIGait(本研究)97.796.390.994.9
Tab.2 Cross-view ablation experiment results of 3D-SIM on CASIA-V dataset
组别R-1/%
NMBGCL均值
基准组90.388.580.686.4
实验组197.796.390.994.9
实验组296.393.587.292.3
实验组396.894.188.593.1
Tab.3 Ablation study results on internal functional decomposition of 3D-SIM module (CASIA-V, single-variable control)
贡献类型ΔR/百分点η/%
3D-SIM总贡献8.5100
真实3D姿态引导的贡献2.630.5
完整特征提取的贡献1.821.1
协同贡献4.148.2
Tab.4 Quantitative analysis results of contribution ratio of each function in 3D-SIM
类别组别R-1/%
NMBGCL均值
轮廓特征基准组89.285.178.384.2
实验组189.785.678.784.7
视频姿态特征基准组82.576.469.776.2
实验组185.781.977.581.7
Tab.5 Verification results of modal information preservation for contour and video pose features of 3D-SIM module (single-modal independent discrimination)
模型$ {N}_{\text{param}}\text{} $/106$ {F}_{\text{flops}}\text{} $/109$ {t}_{\text{inf}}\text{} $/ms$ {v}_{\text{inf}}\text{} $/(帧·s?1)
MSAFF(3D)23.497.1118.83±0.6253.11
3D-SIM0.260.13
SIGait23.757.2419.56±0.5851.12
Tab.7 Computational efficiency comparison of MSAFF(3D)/3D-SIM/SIGait
输入序列长度/帧R-1/%
NMBGCL均值
1692.390.184.789.0
2495.693.888.592.6
3297.796.390.995.0
4097.996.591.295.2
4898.096.691.395.3
Tab.8 Performance comparison under different input sequence lengths
采样间隔/帧实际采样率/(帧·s?1)R-1/%
NMBGCL均值
13097.796.390.994.9
21596.895.289.693.9
31094.592.787.391.5
47.590.288.583.187.3
Tab.9 Performance comparison under different frame sampling rates
训练ID数R-1/%
NMBGCL均值
2089.587.281.386.0
3593.891.585.790.3
5096.294.188.993.1
6597.796.390.994.9
8098.196.891.595.5
Tab.10 Performance comparison under different numbers of training IDs
[1]   MAHMOUD M, KASEM M S, KANG H S A comprehensive survey of masked faces: recognition, detection, and unmasking[J]. Applied Sciences, 2024, 14 (19): 8781
doi: 10.3390/app14198781
[2]   JIA Z, HUANG C, WANG Z, et al Finger recovery transformer: toward better incomplete fingerprint identification[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 8860- 8874
doi: 10.1109/TIFS.2024.3419690
[3]   KUEHLKAMP A, BOYD A, CZAJKA A, et al. Interpretable deep learning-based forensic iris segmentation and recognition [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. Waikoloa: IEEE, 2022: 359–368.
[4]   赵晓东, 刘作军, 陈玲玲, 等 下肢假肢穿戴者跑动步态识别方法[J]. 浙江大学学报: 工学版, 2018, 52 (10): 1980- 1988
ZHAO Xiaodong, LIU Zuojun, CHEN Lingling, et al Approach of running gait recognition for lower limb amputees[J]. Journal of Zhejiang University: Engineering Science, 2018, 52 (10): 1980- 1988
doi: 10.3785/j.issn.1008-973X.2018.10.018
[5]   CHAO H, WANG K, HE Y, et al GaitSet: cross-view gait recognition through utilizing gait as a deep set[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (7): 3467- 3478
doi: 10.1109/tpami.2021.3057879
[6]   FAN C, PENG Y, CAO C, et al. GaitPart: temporal part-based model for gait recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 14213–14221.
[7]   HUANG Z, XUE D, SHEN X, et al. 3D local convolutional neural networks for gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 14900–14909.
[8]   LIN B, ZHANG S, YU X. Gait recognition via effective global-local feature representation and local temporal aggregation [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 14628–14636.
[9]   WU Z, HUANG Y, WANG L, et al A comprehensive study on cross-view gait based human identification with deep CNNs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (2): 209- 226
doi: 10.1109/TPAMI.2016.2545669
[10]   HUANG X, ZHU D, WANG H, et al. Context-sensitive temporal feature learning for gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 12889–12898.
[11]   WANG M, GUO X, LIN B, et al. DyGait: exploiting dynamic representations for high-performance gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 13378–13387.
[12]   LIAO R, YU S, AN W, et al A model-based gait recognition method with body pose and human prior knowledge[J]. Pattern Recognition, 2020, 98: 107069
doi: 10.1016/j.patcog.2019.107069
[13]   TEEPE T, KHAN A, GILG J, et al. Gaitgraph: graph convolutional network for skeleton-based gait recognition [C]// Proceedings of the IEEE International Conference on Image Processing. Anchorage: IEEE, 2021: 2314–2318.
[14]   FU Y, MENG S, HOU S, et al. GPGait: generalized pose-based gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 19538–19547.
[15]   ZHANG C, CHEN X P, HAN G Q, et al Spatial transformer network on skeleton-based gait recognition[J]. Expert Systems, 2023, 40 (6): e13244
doi: 10.1111/exsy.13244
[16]   GUO H, JI Q. Physics-augmented autoencoder for 3D skeleton-based gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 19570–19581.
[17]   吴越, 梁铮, 高巍, 等 基于SMPL模态分解与嵌入融合的多模态步态识别[J]. 浙江大学学报: 工学版, 2026, 60 (1): 52- 60
WU Yue, LIANG Zheng, GAO Wei, et al Multi-modal gait recognition based on SMPL model decomposition and embedding fusion[J]. Journal of Zhejiang University: Engineering Science, 2026, 60 (1): 52- 60
[18]   ZOU S, XIONG J, FAN C, et al. A multi-stage adaptive feature fusion neural network for multimodal gait recognition [C]// Proceedings of the IEEE International Joint Conference on Biometrics. Ljubljana: IEEE, 2024: 1–10.
[19]   LI G, GUO L, ZHANG R, et al TransGait: multimodal-based gait recognition with set transformer[J]. Applied Intelligence, 2023, 53 (2): 1535- 1547
doi: 10.1007/s10489-022-03543-y
[20]   ZHENG W, ZHU H, ZHENG Z, et al GaitSTR: gait recognition with sequential two-stream refinement[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2024, 6 (4): 528- 538
doi: 10.1109/TBIOM.2024.3390626
[21]   FAN C, MA J, JIN D, et al. SkeletonGait: gait recognition using skeleton maps [C]// Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI Press, 2024: 1662–1669.
[22]   YU S, TAN D, TAN T. A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition [C]// Proceedings of the 18th International Conference on Pattern Recognition. Hong Kong: IEEE, 2006: 441–444.
[23]   TAKEMURA N, MAKIHARA Y, MURAMATSU D, et al Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition[J]. IPSJ Transactions on Computer Vision and Applications, 2018, 10 (1): 4
doi: 10.1186/s41074-018-0039-6
[24]   ZHENG J, LIU X, LIU W, et al. Gait recognition in the wild with dense 3D representations and a benchmark [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 20196–20205.
[25]   ZHU Z, GUO X, YANG T, et al. Gait recognition in the wild: a benchmark [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 14769–14779.
[26]   KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. [2025–11–23]. https://arxiv.org/abs/1609.02907.
[27]   LI J, ZHANG Y, SHAN H, et al. Gaitcotr: improved spatial-temporal representation for gait recognition with a hybrid convolution-transformer framework [C]// 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023: 1–5.
[28]   SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 5686–5696.
[29]   PAVLLO D, FEICHTENHOFER C, GRANGIER D, et al. 3D human pose estimation in video with temporal convolutions and semi-supervised training [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 7745–7754.
[30]   LIN S, RYABTSEV A, SENGUPTA S, et al. Real-time high-resolution background matting [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 8758–8767.
[31]   TAKEMURA N, MAKIHARA Y, MURAMATSU D, et al Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition[J]. IPSJ Transactions on Computer Vision and Applications, 2018, 10 (1): 4
doi: 10.1186/s41074-018-0039-6
[1] Kaijun WU,Yunqi ZHENG,Ding WEI,Haixiang YUAN. YOLOv8s based lightweight algorithm for small object detection in aerial imagery[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1912-1923.
[2] Yijie TANG,Mingen ZHONG,Bingan YUAN,Kang FAN,Jiawei TAN,Zhiqiang LIN. Road segmentation algorithm based on joint orthogonal feature fusion and large kernel separable attention[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(9): 1942-1952.
[3] Guangqiu CHEN,Tianrong REN,Jin DUAN,Dandan HUANG. Cross-modal semantic segmentation algorithm with edge-assisted and multi-level feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(8): 1782-1791.
[4] Zheng WANG,Mengjun ZHANG,Nan JIANG,Wanliang WANG,Hangyao TU. Daily runoff prediction using LSTM based on multi-feature fusion and Newton-Raphson-based optimizer[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(7): 1567-1576.
[5] Naizhou ZHANG,Yunchao ZHAO,Wei CAO,Xiaojian ZHANG. Image captioning generation based on multiple-view cross-modal feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(6): 1205-1212.
[6] Bo DONG,Donghao LV,Dahua YU,Xiaowei DU. VAE model combined with multi-domain feature for muscle fatigue analysis[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(6): 1317-1328.
[7] Guoyan LI,Wei YU,Yupeng MEI,Minghui ZHANG,Xinqiang WANG. Building extraction from remote sensing images with global-local feature fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(5): 1100-1108.
[8] Tianhe YU,Wenlong WANG,Yong LIU,Zhuangzhuang YANG,Shanchong HOU. Improved algorithm for identifying occluded vehicles and pedestrians in foggy images[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(4): 738-750.
[9] Long MA,Yongqi HOU,Baijing WU,Li GAO,Jianwei DENG,Guanghui YAN. Water-floating garbage trajectory prediction model based on multi-scale graph convolution[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(4): 751-762.
[10] Xiaoan BAO,Enlin CHEN,Na ZHANG,Xiaomei TU,Biao WU,Qingqi ZHANG. 3D human pose estimation based on multi-scale encoder fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 565-573.
[11] Guoyan LI,Penghui LI,Rong LIU,Yupeng MEI,Minghui ZHANG. Remote sensing road extraction by fusing multi-scale resolution and strip feature[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(3): 585-593.
[12] Jiangang ZHANG,Xiao LI,Dandan FENG. Dynamic kernel perception for pavement distress detection in UAV inspection[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2141-2152.
[13] Hongxia NIU,Dingchao FENG,Tao HOU. Real-time detection algorithm for railway foreign objects in complex weather conditions based on improved RT-DETR[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2165-2175.
[14] Jian XIAO,Xinze HE,Hongliang CHENG,Xiaoyuan YANG,Xin HU. Aerial small target detection algorithm based on multi-scale feature enhancement[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 19-31.
[15] Yue WU,Zheng LIANG,Wei GAO,Maoda YANG,Peisen ZHAO,Hongxia DENG,Yuanyuan CHANG. Multi-modal gait recognition based on SMPL model decomposition and embedding fusion[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(1): 52-60.