Please wait a minute...
浙江大学学报(工学版)  2026, Vol. 60 Issue (10): 2196-2206    DOI: 10.3785/j.issn.1008-973X.2026.10.012
计算机技术与控制工程     
基于姿态诱导的多模态步态识别
高巍1(),于志栋1,常媛媛2,苗旺1,尹永亮2,邓红霞1,*()
1. 太原理工大学 计算机科学与技术学院(大数据学院),山西 太原 030024
2. 太原理工大学 体育与健康工程学院,山西 太原 030024
Multimodal gait recognition based on pose induction
Wei GAO1(),Zhidong YU1,Yuanyuan CHANG2,Wang MIAO1,Yongliang YIN2,Hongxia DENG1,*()
1. College of Computer Science and Technology (College of Data Science), Taiyuan University of Technology, Taiyuan 030024, China
2. School of Physical Education and Health Engineering, Taiyuan University of Technology, Taiyuan 030024, China
 全文: PDF(1270 KB)   HTML
摘要:

为了实现复杂环境下步态的多视角细粒度识别,提出基于姿态诱导的多模态步态识别方法,并构建包含真实三维姿态的多模态数据集(CASIA-V). 该数据集通过多设备同步采集93名受试者在正常、背包、穿大衣行走下的11视角数据,包含11253个RGB序列及对应动捕姿态与深度图. 在方法层面,提出时空对齐策略实现动捕与视频姿态匹配,设计可独立训练的三维姿态诱导模块(3D-SIM)提取真实姿态语义,并构建双分支模型SIGait融合姿态与轮廓特征. 实验表明,在CASIA-V数据集的跨视角评估协议下(排除相同视角样本对),SIGait在正常行走场景下的Rank-1准确率达97.7%;相较基线方法MSAFF,在背包与穿大衣场景下Rank-1准确率分别提升5.1和7.1个百分点,且推理过程无需动捕数据,验证了真实三维姿态对复杂场景步态识别的有效促进. 结果同时显示,真实三维姿态语义能增强多模态特征表达,使识别系统在视角变化与外观干扰条件下保持稳定识别性能.

关键词: 步态识别多模态数据集三维姿态时空对齐姿态诱导特征融合惯性动捕    
Abstract:

A pose-induced multimodal gait recognition method was proposed and a new multimodal dataset named CASIA-V containing real 3D human poses was constructed, in order to achieve multi-view fine-grained gait recognition in complex environments. The dataset was collected using multiple synchronized devices, capturing 93 subjects walking under three conditions (normal, carrying a bag, and wearing a coat) from 11 different viewpoints. It contained 11 253 RGB sequences along with corresponding motion-capture poses and depth maps. At the methodological level, a spatiotemporal alignment strategy was introduced to match motion-capture poses with video-based poses. A separately trainable 3D Pose-Induced Module (3D-SIM) was designed to extract real pose semantics, and a dual-branch model named SIGait was constructed to fuse pose and silhouette features. The experiment showed that, under the cross-view evaluation protocol of the CASIA-V dataset (excluding samples from the same view), SIGait achieved a Rank-1 accuracy of 97.7% in the normal walking scenario. Compared with the baseline method MSAFF, SIGait improved the Rank-1 accuracy by 5.1 and 7.1 percentage points in the carrying-bag and wearing-coat scenarios, respectively. Notably, the inference process did not require motion-capture data, validating that real 3D pose effectively enhanced gait recognition in complex scenarios. The results further showed that real 3D pose semantics enhanced multimodal feature representation and maintained stable recognition performance under view variations and appearance interference.

Key words: gait recognition    multi-modal dataset    3D pose    spatiotemporal alignment    pose induction    feature fusion    inertial motion capture
收稿日期: 2025-12-16 出版日期: 2026-07-28
CLC:  TP 393  
基金资助: 山西省中央引导地方科技发展资金项目(YDZJSX2022A016);山西省重点研发计划资助项目(2022ZDYF128);山西省科技战略项目(202404030401080).
通讯作者: 邓红霞     E-mail: gw190428@163.com;denghongxia@tyut.edu.cn
作者简介: 高巍(1999—),男,硕士生,从事步态识别研究. orcid.org/0009-0004-7603-4909. E-mail:gw190428@163.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
高巍
于志栋
常媛媛
苗旺
尹永亮
邓红霞

引用本文:

高巍,于志栋,常媛媛,苗旺,尹永亮,邓红霞. 基于姿态诱导的多模态步态识别[J]. 浙江大学学报(工学版), 2026, 60(10): 2196-2206.

Wei GAO,Zhidong YU,Yuanyuan CHANG,Wang MIAO,Yongliang YIN,Hongxia DENG. Multimodal gait recognition based on pose induction. Journal of ZheJiang University (Engineering Science), 2026, 60(10): 2196-2206.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2026.10.012        https://www.zjujournals.com/eng/CN/Y2026/V60/I10/2196

图 1  数据采集场地示意图
图 2  时间对齐示意图
图 3  空间对齐示意图
图 4  CASIA-V中不同步态数据的示例
图 5  CASIA-V中年龄、性别和序列长度的分布
图 6  SIGait网络结构
图 7  3D-SIM 网络结构
池化策略R-1/%
NMBGCL均值
最大池化96.895.189.793.9
全局池化97.295.790.294.4
平均池化97.796.390.994.9
表 6  不同池化策略的性能对比
方法R-1/%
NMBGCL均值
GaitGraph[13]84.774.866.375.2
PoseGait[12]69.745.137.250.6
GaitSet[5]92.887.269.983.3
TransGait[19]96.494.485.392.0
SkeletonGait++[21]95.990.984.790.5
MSAFF[18]94.691.283.889.8
SIGait(本研究)97.796.390.994.9
表 1  不同方法在CASIA-V数据集上的跨视角对比结果
方法R-1/%
NMBGCL均值
GaitGraph[13]84.774.866.375.2
GaitGraph(3D)69.059.653.360.6
GaitGraph(3D)+89.484.075.883.0
PoseGait(3D)[12]68.744.536.049.7
PoseGait(3D)+91.184.977.884.6
MSAFF[18]94.691.283.889.8
MSAFF(3D)90.388.580.686.4
SIGait(本研究)97.796.390.994.9
表 2  在CASIA-V数据集上对3D-SIM的跨视角消融实验结果
组别R-1/%
NMBGCL均值
基准组90.388.580.686.4
实验组197.796.390.994.9
实验组296.393.587.292.3
实验组396.894.188.593.1
表 3  3D-SIM模块内部功能拆解的消融实验结果(CASIA-V,单一变量控制)
贡献类型ΔR/百分点η/%
3D-SIM总贡献8.5100
真实3D姿态引导的贡献2.630.5
完整特征提取的贡献1.821.1
协同贡献4.148.2
表 4  3D-SIM 模块各功能贡献比例量化分析结果
类别组别R-1/%
NMBGCL均值
轮廓特征基准组89.285.178.384.2
实验组189.785.678.784.7
视频姿态特征基准组82.576.469.776.2
实验组185.781.977.581.7
表 5  3D-SIM模块对轮廓与视频姿态特征的模态信息保留验证结果(单模态单独判别)
模型$ {N}_{\text{param}}\text{} $/106$ {F}_{\text{flops}}\text{} $/109$ {t}_{\text{inf}}\text{} $/ms$ {v}_{\text{inf}}\text{} $/(帧·s?1)
MSAFF(3D)23.497.1118.83±0.6253.11
3D-SIM0.260.13
SIGait23.757.2419.56±0.5851.12
表 7  MSAFF(3D)/3D-SIM/SIGait计算效率对比
输入序列长度/帧R-1/%
NMBGCL均值
1692.390.184.789.0
2495.693.888.592.6
3297.796.390.995.0
4097.996.591.295.2
4898.096.691.395.3
表 8  不同输入序列长度下的性能对比
采样间隔/帧实际采样率/(帧·s?1)R-1/%
NMBGCL均值
13097.796.390.994.9
21596.895.289.693.9
31094.592.787.391.5
47.590.288.583.187.3
表 9  不同帧采样率下的性能对比
训练ID数R-1/%
NMBGCL均值
2089.587.281.386.0
3593.891.585.790.3
5096.294.188.993.1
6597.796.390.994.9
8098.196.891.595.5
表 10  不同训练ID数下的性能对比
1 MAHMOUD M, KASEM M S, KANG H S A comprehensive survey of masked faces: recognition, detection, and unmasking[J]. Applied Sciences, 2024, 14 (19): 8781
doi: 10.3390/app14198781
2 JIA Z, HUANG C, WANG Z, et al Finger recovery transformer: toward better incomplete fingerprint identification[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 8860- 8874
doi: 10.1109/TIFS.2024.3419690
3 KUEHLKAMP A, BOYD A, CZAJKA A, et al. Interpretable deep learning-based forensic iris segmentation and recognition [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. Waikoloa: IEEE, 2022: 359–368.
4 赵晓东, 刘作军, 陈玲玲, 等 下肢假肢穿戴者跑动步态识别方法[J]. 浙江大学学报: 工学版, 2018, 52 (10): 1980- 1988
ZHAO Xiaodong, LIU Zuojun, CHEN Lingling, et al Approach of running gait recognition for lower limb amputees[J]. Journal of Zhejiang University: Engineering Science, 2018, 52 (10): 1980- 1988
doi: 10.3785/j.issn.1008-973X.2018.10.018
5 CHAO H, WANG K, HE Y, et al GaitSet: cross-view gait recognition through utilizing gait as a deep set[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (7): 3467- 3478
doi: 10.1109/tpami.2021.3057879
6 FAN C, PENG Y, CAO C, et al. GaitPart: temporal part-based model for gait recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 14213–14221.
7 HUANG Z, XUE D, SHEN X, et al. 3D local convolutional neural networks for gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 14900–14909.
8 LIN B, ZHANG S, YU X. Gait recognition via effective global-local feature representation and local temporal aggregation [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 14628–14636.
9 WU Z, HUANG Y, WANG L, et al A comprehensive study on cross-view gait based human identification with deep CNNs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (2): 209- 226
doi: 10.1109/TPAMI.2016.2545669
10 HUANG X, ZHU D, WANG H, et al. Context-sensitive temporal feature learning for gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 12889–12898.
11 WANG M, GUO X, LIN B, et al. DyGait: exploiting dynamic representations for high-performance gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 13378–13387.
12 LIAO R, YU S, AN W, et al A model-based gait recognition method with body pose and human prior knowledge[J]. Pattern Recognition, 2020, 98: 107069
doi: 10.1016/j.patcog.2019.107069
13 TEEPE T, KHAN A, GILG J, et al. Gaitgraph: graph convolutional network for skeleton-based gait recognition [C]// Proceedings of the IEEE International Conference on Image Processing. Anchorage: IEEE, 2021: 2314–2318.
14 FU Y, MENG S, HOU S, et al. GPGait: generalized pose-based gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 19538–19547.
15 ZHANG C, CHEN X P, HAN G Q, et al Spatial transformer network on skeleton-based gait recognition[J]. Expert Systems, 2023, 40 (6): e13244
doi: 10.1111/exsy.13244
16 GUO H, JI Q. Physics-augmented autoencoder for 3D skeleton-based gait recognition [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 19570–19581.
17 吴越, 梁铮, 高巍, 等 基于SMPL模态分解与嵌入融合的多模态步态识别[J]. 浙江大学学报: 工学版, 2026, 60 (1): 52- 60
WU Yue, LIANG Zheng, GAO Wei, et al Multi-modal gait recognition based on SMPL model decomposition and embedding fusion[J]. Journal of Zhejiang University: Engineering Science, 2026, 60 (1): 52- 60
18 ZOU S, XIONG J, FAN C, et al. A multi-stage adaptive feature fusion neural network for multimodal gait recognition [C]// Proceedings of the IEEE International Joint Conference on Biometrics. Ljubljana: IEEE, 2024: 1–10.
19 LI G, GUO L, ZHANG R, et al TransGait: multimodal-based gait recognition with set transformer[J]. Applied Intelligence, 2023, 53 (2): 1535- 1547
doi: 10.1007/s10489-022-03543-y
20 ZHENG W, ZHU H, ZHENG Z, et al GaitSTR: gait recognition with sequential two-stream refinement[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2024, 6 (4): 528- 538
doi: 10.1109/TBIOM.2024.3390626
21 FAN C, MA J, JIN D, et al. SkeletonGait: gait recognition using skeleton maps [C]// Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI Press, 2024: 1662–1669.
22 YU S, TAN D, TAN T. A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition [C]// Proceedings of the 18th International Conference on Pattern Recognition. Hong Kong: IEEE, 2006: 441–444.
23 TAKEMURA N, MAKIHARA Y, MURAMATSU D, et al Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition[J]. IPSJ Transactions on Computer Vision and Applications, 2018, 10 (1): 4
doi: 10.1186/s41074-018-0039-6
24 ZHENG J, LIU X, LIU W, et al. Gait recognition in the wild with dense 3D representations and a benchmark [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 20196–20205.
25 ZHU Z, GUO X, YANG T, et al. Gait recognition in the wild: a benchmark [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 14769–14779.
26 KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. [2025–11–23]. https://arxiv.org/abs/1609.02907.
27 LI J, ZHANG Y, SHAN H, et al. Gaitcotr: improved spatial-temporal representation for gait recognition with a hybrid convolution-transformer framework [C]// 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023: 1–5.
28 SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 5686–5696.
29 PAVLLO D, FEICHTENHOFER C, GRANGIER D, et al. 3D human pose estimation in video with temporal convolutions and semi-supervised training [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 7745–7754.
30 LIN S, RYABTSEV A, SENGUPTA S, et al. Real-time high-resolution background matting [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 8758–8767.
31 TAKEMURA N, MAKIHARA Y, MURAMATSU D, et al Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition[J]. IPSJ Transactions on Computer Vision and Applications, 2018, 10 (1): 4
doi: 10.1186/s41074-018-0039-6
[1] 邬开俊,郑云琦,魏鼎,袁海翔. 基于YOLOv8s的轻量化航拍图像小目标检测算法[J]. 浙江大学学报(工学版), 2026, 60(9): 1912-1923.
[2] 汤毅杰,钟铭恩,袁彬淦,范康,谭佳威,林志强. 联合正交特征融合与大核可分离注意力的道路分割算法[J]. 浙江大学学报(工学版), 2026, 60(9): 1942-1952.
[3] 陈广秋,任天蓉,段锦,黄丹丹. 结合边缘辅助与多级特征融合的跨模态语义分割算法[J]. 浙江大学学报(工学版), 2026, 60(8): 1782-1791.
[4] 王铮,张梦君,姜楠,王万良,屠杭垚. 基于多特征融合和牛顿-拉夫逊优化算法的LSTM日径流预测[J]. 浙江大学学报(工学版), 2026, 60(7): 1567-1576.
[5] 张乃洲,赵云超,曹薇,张啸剑. 基于多视图跨模态特征融合的图像描述生成[J]. 浙江大学学报(工学版), 2026, 60(6): 1205-1212.
[6] 董博,吕东澔,喻大华,杜晓炜. 融合多域特征的VAE模型在肌肉疲劳分析中的应用[J]. 浙江大学学报(工学版), 2026, 60(6): 1317-1328.
[7] 李国燕,于威,梅玉鹏,张明辉,王新强. 全局局部特征融合的遥感图像建筑物提取[J]. 浙江大学学报(工学版), 2026, 60(5): 1100-1108.
[8] 于天河,王文龙,刘镛,杨壮壮,侯善冲. 改进的有雾图像中被遮挡车辆及行人识别算法[J]. 浙江大学学报(工学版), 2026, 60(4): 738-750.
[9] 马龙,候永琪,吴佰靖,高丽,邓建伟,闫光辉. 多尺度图卷积下的水漂垃圾轨迹预测模型[J]. 浙江大学学报(工学版), 2026, 60(4): 751-762.
[10] 包晓安,陈恩琳,张娜,涂小妹,吴彪,张庆琪. 基于多尺度编码器融合的三维人体姿态估计算法[J]. 浙江大学学报(工学版), 2026, 60(3): 565-573.
[11] 李国燕,李鹏辉,刘榕,梅玉鹏,张明辉. 融合多尺度分辨率和带状特征的遥感道路提取[J]. 浙江大学学报(工学版), 2026, 60(3): 585-593.
[12] 张建刚,李肖,冯丹丹. 基于动态核感知的无人机视角路面病害检测方法[J]. 浙江大学学报(工学版), 2026, 60(10): 2141-2152.
[13] 牛宏侠,冯鼎超,侯涛. 基于改进RT-DETR的复杂天气下铁路异物实时检测算法[J]. 浙江大学学报(工学版), 2026, 60(10): 2165-2175.
[14] 肖剑,何昕泽,程鸿亮,杨小苑,胡欣. 基于多尺度特征增强的航拍小目标检测算法[J]. 浙江大学学报(工学版), 2026, 60(1): 19-31.
[15] 吴越,梁铮,高巍,杨茂达,赵培森,邓红霞,常媛媛. 基于SMPL模态分解与嵌入融合的多模态步态识别[J]. 浙江大学学报(工学版), 2026, 60(1): 52-60.