Please wait a minute...
浙江大学学报(工学版)
浙江大学学报(工学版)  2021, Vol. 55 Issue (1): 153-161    DOI: 10.3785/j.issn.1008-973X.2021.01.018
机械工程     
由LeNet-5从单张着装图像重建三维人体
许豪灿1,2(),李基拓1,2,*(),陆国栋1,2
1. 浙江大学 机械工程学院，浙江 杭州 310027
2. 浙江大学 机器人研究院，浙江 余姚 315400
Reconstruction of three-dimensional human bodies from single image by LeNet-5
Hao-can XU1,2(),Ji-tuo LI1,2,*(),Guo-dong LU1,2
1. School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China
2. Robotics Institute, Zhejiang University, Yuyao 315400, China
 全文: PDF(1160 KB)   HTML
摘要：

提出基于LeNet-5的从单张着装图像恢复人体三维形状的方法，建立着装人体正面轮廓和人体形状空间之间的映射模型，实现了高效、精确的三维人体建模，可以应用于对人体表面形状精度要求较高的场合，如虚拟试衣. 基于PGA在流型空间上对公开的三维人体数据集进行数据扩增，给虚拟人体进行着装，构建着装人体数据库. 从着装人体正面投影图像中提取信息，以人体形状参数及正、侧面轮廓信息为约束，基于LeNet-5完成三维人体重建. 实验证明，对于身穿不同款式服装的人，采用的模型通常都能从单张着装图像中重建得到较高精度的三维人体模型.
关键词： 三维人体重建;  虚拟试衣;  数据扩增;  着装人体;  深度学习    
Abstract:

A novel human body modeling method that can reconstruct three-dimensional (3D) human bodies from single dressed human body image based on LeNet-5 was proposed. The method can reconstruct 3D human bodies accurately and efficiently, and the reconstruction results can be potentially used in some occasions where require precise surface shapes, such as virtual try-on systems. 3D human bodies collected from open datasets were selected and augmented on manifolds with PGA. A dressed human body database was established after dressing these 3D human bodies with virtual garments in various types and sizes. Feature descriptors were extracted from the frontal projected images of dressed human bodies. The corresponding 3D human bodies were constructed through LeNet-5 with the constraints of shape parameters as well as the frontal and lateral contours. The experimental results show that the model can reconstruct a high-precision 3D human body from a single dressed human body image for people wearing different styles of clothing.
Key words: three-dimensional human modeling    virtual try-on    data augmentation    dressed human body    deep learning
收稿日期: 2020-01-09 出版日期: 2021-01-05
CLC:  TP 399  
基金资助: 国家重点研发计划资助项目（2018YFB1700704）；国家自然科学基金资助项目（61732015）；中央高校基本科研业务费专项资助项目（2019QNA4001）；浙江省自然科学基金资助项目（LY18F020004）
通讯作者: 李基拓     E-mail: haocan_xu@zju.edu.cn;jituo_li@zju.edu.cn
作者简介: 许豪灿（1993—），男，博士生，从事计算机图形学的研究. orcid.org/0000-0002-1474-7039. E-mail： haocan_xu@zju.edu.cn
服务　　
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章　　
许豪灿
李基拓
陆国栋

引用本文:

许豪灿,李基拓,陆国栋. 由LeNet-5从单张着装图像重建三维人体[J]. 浙江大学学报(工学版), 2021, 55(1): 153-161.

Hao-can XU,Ji-tuo LI,Guo-dong LU. Reconstruction of three-dimensional human bodies from single image by LeNet-5. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 153-161.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2021.01.018    或    http://www.zjujournals.com/eng/CN/Y2021/V55/I1/153

图 1  由单张图像重建三维人体流程
图 2  人体数据集扩增
图 3  在虚拟人体表面添加服装
图 4  LeNet-5网络结构
损失函数 e /cm
总体误差 胸围 腰围 臀围 手长 腿长
${L_\gamma } $ 1.76 3.27 3.18 3.51 1.94 2.04
${L_\gamma } + \varphi {L_{\rm{f}}} $ 1.36 2.34 2.49 2.72 1.48 1.59
${L_\gamma } $ + φLs 1.31 2.36 2.23 2.66 1.41 1.62
Ltotal 1.15 1.97 2.08 2.32 1.21 1.45
表 1  不同损失函数下重建结果误差
方法 e /cm
总体误差 胸围 腰围 臀围
LeNet-5 1.15 1.97 2.08 2.32
文献[15]方法 ? 0.1~4.5 0.6~3.4 ?
文献[12]方法 7.59 ? ? ?
文献[14]方法 5.68 ? ? ?
文献[16]方法 5.99 ? ? ?
表 2  不同方法重建结果误差
人体姿态 e /cm
总体误差 胸围 腰围 臀围 手长 腿长
P = 15° 1.56 3.45 2.91 3.34 1.84 2.15
P = 25° 1.26 2.06 2.24 2.29 1.41 1.53
P = 30° 1.15 1.97 2.08 2.32 1.21 1.45
P = 35° 1.19 1.95 2.17 2.57 1.15 1.57
P = 45° 1.69 3.16 3.05 3.40 2.02 2.01
P = 90° 2.68 4.52 5.05 5.13 3.17 3.64
L = 0 1.21 2.14 2.09 2.45 1.36 1.68
表 3  不同姿态下人体的误差
图 5  不同姿态下的三维人体
图 6  不同体型的三维人体重建
图 7  人体正面图像预处理
人体 着装 e /cm
胸围 腰围 臀围 手长 腿长
人体1 长衣长裤 2.35 2.08 1.14 1.96 0.91
人体1 短裙 3.15 2.93 3.20 1.69 1.46
人体2 长衣长裤 1.05 0.81 1.02 0.36 1.34
人体2 短裙 1.47 1.61 1.84 1.37 2.11
人体3 长衣长裤 1.54 1.72 3.03 2.35 1.71
人体3 短裙 2.61 2.20 2.61 3.22 1.68
表 4  不同人体的重建误差
图 8  从正面图像恢复三维人体，姿态A，长衣长裤
图 9  从正面图像恢复三维人体，姿态A，着短裙
人体姿态 e /cm
胸围 腰围 臀围 手长 腿长
P = 15° 1.84 0.73 1.51 1.46 1.58
P = 25° 1.62 0.88 0.67 0.93 1.61
P = 30° 1.05 0.81 1.02 0.73 1.34
P = 35° 0.91 1.20 1.53 0.36 0.95
P = 45° 2.15 2.06 2.07 1.04 1.36
P = 90° 2.16 3.84 4.21 5.13 4.38
L = 0 1.26 1.39 0.79 0.51 1.37
表 5  不同姿态人体的重建误差
图 10  从不同姿态人体图像恢复三维人体
服装款式 e /cm
胸围 腰围 臀围 手长 腿长
款式1 0.73 1.43 2.71 1.82 1.51
款式2 2.65 2.81 1.97 1.60 0.82
款式3 4.79 4.56 1.76 3.14 1.01
表 6  不同服装下人体的重建误差
图 11  从同一人体不同着装的图像恢复三维人体
1 ALLDIECK T, MAGNOR M, XU W, et al. Detailed human avatars from monocular video [C]// International Conference on 3D Vision. Verona: IEEE, 2018: 98-109.
2 TONG J, ZHOU J, LIU L, et al Scanning 3D full human bodies using kinects[J]. IEEE Transactions on Visualization and Computer Graphics, 2012, 18 (4): 643- 650
doi: 10.1109/TVCG.2012.56
3 CHEN G, LI J, WANG B, et al Reconstructing 3D human models with a kinect[J]. Computer Animation and Virtual Worlds, 2016, 27 (1): 72- 85
doi: 10.1002/cav.1632
4 CHEN G, LI J, ZENG J, et al Optimizing human model reconstruction from RGB-D image based on skin detection[J]. Virtual Reality, 2016, 20 (3): 159- 172
doi: 10.1007/s10055-016-0291-y
5 WEISS A, HIRSHBERG D, BLACL M J. Home 3D body scans from noisy image and range data [C]// International Conference on Computer Vision. Barcelona: IEEE, 2011: 1951-1958.
6 WANG C C L Parameterization and parametric design of mannequins[J]. Computer-Aided Design, 2005, 37 (1): 83- 98
doi: 10.1016/j.cad.2004.05.001
7 BEAK S Y, LEE K Parametric human body shape modeling framework for human-centered product design[J]. Computer-Aided Design, 2012, 44 (1): 56- 67
doi: 10.1016/j.cad.2010.12.006
8 HUANG J, KWOK T H, ZHOU C Parametric design for human body modeling by wireframe-assisted deep learning[J]. Computer-Aided Design, 2019, 108: 19- 29
doi: 10.1016/j.cad.2018.10.004
9 ANGUELOV D, SRINIVASAN P, KOLLER D, et al SCAPE: shape completion and animation of people[J]. ACM Transactions on Graphics, 2005, 24 (3): 408- 416
doi: 10.1145/1073204.1073207
10 LOPER M, MAHMOOD N, ROMERO J, et al SMPL: a skinned multi-person linear model[J]. ACM Transactions on Graphics, 2015, 34 (6): 248
11 POPA A I, ZANFIR M, SMINCHISESCU C. Deep multitask architecture for integrated 2d and 3d human sensing [C]//Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 6289-6298.
12 PAVLAKOS G, ZHU L, ZHOU X, et al. Learning to estimate 3D human pose and shape from a single color image [C]// Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 459-468.
13 JI Z, QI X, WANG Y, et al Human body shape reconstruction from binary silhouette images[J]. Computer Aided Geometric Design, 2019, 71: 231- 243
doi: 10.1016/j.cagd.2019.04.019
14 KANAZAWA A, BLACK M J, JACOBS D W, et al. End-to-end recovery of human shape and pose [C]// Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7122-7131.
15 GUAN P, WEISS A, BALAN A O, et al. Estimating human shape and pose from a single image [C]// International Conference on Computer Vision. Florida: IEEE, 2009: 1381-1388.
16 OMRAN M, LASSNER C, PONS-MOLL G, et al. Neural body fitting: unifying deep learning and model based human pose and shape estimation [C]// International Conference on 3D Vision. Verona: IEEE, 2018: 484-494.
17 JOHNSON S, EVERINGHAM M. Clustered pose and nonlinear appearance models for human pose estimation [C]// British Machine Vision Conference. Aberystwyth: BMVA, 2010: 5.
18 IONESCU C, PAPAVA D, OLARU V, et al Human3.6m: large scale datasets and predictive methods for 3d human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36 (7): 1325- 1339
19 LASSNER C, ROMERO J, KIEFEL M, et al. Unite the people: closing the loop between 3d and 2d human representations [C]// Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 6050-6059.
20 PISHCHULIN L, WUHRER S, HELTEN T, et al Building statistical shape space for 3d human modeling[J]. Patten Recognition, 2017, 67: 276- 286
doi: 10.1016/j.patcog.2017.02.018
21 LI J, LU G Customizing 3D garments based on volumetric deformation[J]. Computers in Industry, 2011, 62 (7): 693- 707
doi: 10.1016/j.compind.2011.04.002
22 FREIFELD O, BLACK M J. Lie bodies: a manifold representation of 3D human shape [C]// European Conference on Computer Vision. Berlin: Springer, 2012: 1-14.
23 FLETCHER P T, LU C, JOSHI S. Statistics of shape via principal geodesic analysis on lie groups [C]// IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Wisconsin: IEEE, 2003: 95-101.
24 LECUN Y, BOTTOU L, BENGIO Y, et al Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86 (11): 2278- 2324
doi: 10.1109/5.726791
25 HINTON G E, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors [J]. Computer Science, 2012, 3(4): 212-223.
[1] 许佳辉,王敬昌,陈岭,吴勇. 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报(工学版), 2021, 55(4): 601-607.
[2] 王虹力,郭斌,刘思聪,刘佳琪,仵允港,於志文. 边端融合的终端情境自适应深度感知模型[J]. 浙江大学学报(工学版), 2021, 55(4): 626-638.
[3] 张腾,蒋鑫龙,陈益强,陈前,米涛免,陈彪. 基于腕部姿态的帕金森病用药后开-关期检测[J]. 浙江大学学报(工学版), 2021, 55(4): 639-647.
[4] 徐利锋,黄海帆,丁维龙,范玉雷. 基于改进DenseNet的水果小目标检测[J]. 浙江大学学报(工学版), 2021, 55(2): 377-385.
[5] 黄毅鹏,胡冀苏,钱旭升,周志勇,赵文露,马麒,沈钧康,戴亚康. SE-Mask-RCNN：多参数MRI前列腺癌分割方法[J]. 浙江大学学报(工学版), 2021, 55(1): 203-212.
[6] 郑浦,白宏阳,李伟,郭宏伟. 复杂背景下的小目标检测算法[J]. 浙江大学学报(工学版), 2020, 54(9): 1777-1784.
[7] 陈巧红,陈翊,李文书,贾宇波. 多尺度SE-Xception服装图像分类[J]. 浙江大学学报(工学版), 2020, 54(9): 1727-1735.
[8] 周登文,田金月,马路遥,孙秀秀. 基于多级特征并联的轻量级图像语义分割[J]. 浙江大学学报(工学版), 2020, 54(8): 1516-1524.
[9] 明涛,王丹,郭继昌,李锵. 基于多尺度通道重校准的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2020, 54(7): 1289-1297.
[10] 闫旭,范晓亮,郑传潘,臧彧,王程,程明,陈龙彪. 基于图卷积神经网络的城市交通态势预测算法[J]. 浙江大学学报(工学版), 2020, 54(6): 1147-1155.
[11] 汪周飞,袁伟娜. 基于深度学习的多载波系统信道估计与检测[J]. 浙江大学学报(工学版), 2020, 54(4): 732-738.
[12] 杨冰,莫文博,姚金良. 融合局部特征与深度学习的三维掌纹识别[J]. 浙江大学学报(工学版), 2020, 54(3): 540-545.
[13] 洪炎佳,孟铁豹,黎浩江,刘立志,李立,徐硕瑀,郭圣文. 多模态多维信息融合的鼻咽癌MR图像肿瘤深度分割方法[J]. 浙江大学学报(工学版), 2020, 54(3): 566-573.
[14] 贾子钰,林友芳,张宏钧,王晶. 基于深度卷积神经网络的睡眠分期模型[J]. 浙江大学学报(工学版), 2020, 54(10): 1899-1905.
[15] 王万良,杨小涵,赵燕伟,高楠,吕闯,张兆娟. 采用卷积自编码器网络的图像增强算法[J]. 浙江大学学报(工学版), 2019, 53(9): 1728-1740.