Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2021, Vol. 55 Issue (1): 153-161    DOI: 10.3785/j.issn.1008-973X.2021.01.018
    
Reconstruction of three-dimensional human bodies from single image by LeNet-5
Hao-can XU1,2(),Ji-tuo LI1,2,*(),Guo-dong LU1,2
1. School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China
2. Robotics Institute, Zhejiang University, Yuyao 315400, China
Download: HTML     PDF(1160KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A novel human body modeling method that can reconstruct three-dimensional (3D) human bodies from single dressed human body image based on LeNet-5 was proposed. The method can reconstruct 3D human bodies accurately and efficiently, and the reconstruction results can be potentially used in some occasions where require precise surface shapes, such as virtual try-on systems. 3D human bodies collected from open datasets were selected and augmented on manifolds with PGA. A dressed human body database was established after dressing these 3D human bodies with virtual garments in various types and sizes. Feature descriptors were extracted from the frontal projected images of dressed human bodies. The corresponding 3D human bodies were constructed through LeNet-5 with the constraints of shape parameters as well as the frontal and lateral contours. The experimental results show that the model can reconstruct a high-precision 3D human body from a single dressed human body image for people wearing different styles of clothing.



Key wordsthree-dimensional human modeling      virtual try-on      data augmentation      dressed human body      deep learning     
Received: 09 January 2020      Published: 05 January 2021
CLC:  TP 399  
Corresponding Authors: Ji-tuo LI     E-mail: haocan_xu@zju.edu.cn;jituo_li@zju.edu.cn
Cite this article:

Hao-can XU,Ji-tuo LI,Guo-dong LU. Reconstruction of three-dimensional human bodies from single image by LeNet-5. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 153-161.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2021.01.018     OR     http://www.zjujournals.com/eng/Y2021/V55/I1/153


由LeNet-5从单张着装图像重建三维人体

提出基于LeNet-5的从单张着装图像恢复人体三维形状的方法,建立着装人体正面轮廓和人体形状空间之间的映射模型,实现了高效、精确的三维人体建模,可以应用于对人体表面形状精度要求较高的场合,如虚拟试衣. 基于PGA在流型空间上对公开的三维人体数据集进行数据扩增,给虚拟人体进行着装,构建着装人体数据库. 从着装人体正面投影图像中提取信息,以人体形状参数及正、侧面轮廓信息为约束,基于LeNet-5完成三维人体重建. 实验证明,对于身穿不同款式服装的人,采用的模型通常都能从单张着装图像中重建得到较高精度的三维人体模型.


关键词: 三维人体重建,  虚拟试衣,  数据扩增,  着装人体,  深度学习 
Fig.1 Pipeline of reconstruction of three-dimensional human bodies from single image
Fig.2 Human data augmentation
Fig.3 Garment simulation on human body surface
Fig.4 Network structure of LeNet-5
损失函数 e /cm
总体误差 胸围 腰围 臀围 手长 腿长
${L_\gamma } $ 1.76 3.27 3.18 3.51 1.94 2.04
${L_\gamma } + \varphi {L_{\rm{f}}} $ 1.36 2.34 2.49 2.72 1.48 1.59
${L_\gamma } $ + φLs 1.31 2.36 2.23 2.66 1.41 1.62
Ltotal 1.15 1.97 2.08 2.32 1.21 1.45
Tab.1 Reconstruction error with different loss function
方法 e /cm
总体误差 胸围 腰围 臀围
LeNet-5 1.15 1.97 2.08 2.32
文献[15]方法 ? 0.1~4.5 0.6~3.4 ?
文献[12]方法 7.59 ? ? ?
文献[14]方法 5.68 ? ? ?
文献[16]方法 5.99 ? ? ?
Tab.2 Reconstruction error with different methods
人体姿态 e /cm
总体误差 胸围 腰围 臀围 手长 腿长
P = 15° 1.56 3.45 2.91 3.34 1.84 2.15
P = 25° 1.26 2.06 2.24 2.29 1.41 1.53
P = 30° 1.15 1.97 2.08 2.32 1.21 1.45
P = 35° 1.19 1.95 2.17 2.57 1.15 1.57
P = 45° 1.69 3.16 3.05 3.40 2.02 2.01
P = 90° 2.68 4.52 5.05 5.13 3.17 3.64
L = 0 1.21 2.14 2.09 2.45 1.36 1.68
Tab.3 Reconstruction error on different postures
Fig.5 3D human bodies with different postures
Fig.6 3D human body reconstruction in different shapes
Fig.7 Preprocessing for frontal image
人体 着装 e /cm
胸围 腰围 臀围 手长 腿长
人体1 长衣长裤 2.35 2.08 1.14 1.96 0.91
人体1 短裙 3.15 2.93 3.20 1.69 1.46
人体2 长衣长裤 1.05 0.81 1.02 0.36 1.34
人体2 短裙 1.47 1.61 1.84 1.37 2.11
人体3 长衣长裤 1.54 1.72 3.03 2.35 1.71
人体3 短裙 2.61 2.20 2.61 3.22 1.68
Tab.4 Reconstruction error on different bodies
Fig.8 3D human body reconstruction from images captured in front view, posture A, and long trousers
Fig.9 3D human body reconstruction from images captured in front view, posture A, and short skirt
人体姿态 e /cm
胸围 腰围 臀围 手长 腿长
P = 15° 1.84 0.73 1.51 1.46 1.58
P = 25° 1.62 0.88 0.67 0.93 1.61
P = 30° 1.05 0.81 1.02 0.73 1.34
P = 35° 0.91 1.20 1.53 0.36 0.95
P = 45° 2.15 2.06 2.07 1.04 1.36
P = 90° 2.16 3.84 4.21 5.13 4.38
L = 0 1.26 1.39 0.79 0.51 1.37
Tab.5 Reconstruction error for real human bodies with different postures
Fig.10 Reconstruction from images of human bodies in different postures
服装款式 e /cm
胸围 腰围 臀围 手长 腿长
款式1 0.73 1.43 2.71 1.82 1.51
款式2 2.65 2.81 1.97 1.60 0.82
款式3 4.79 4.56 1.76 3.14 1.01
Tab.6 Reconstruction error for real human bodies with different garments
Fig.11 Reconstruction from images of one human body in different garments
[1]   ALLDIECK T, MAGNOR M, XU W, et al. Detailed human avatars from monocular video [C]// International Conference on 3D Vision. Verona: IEEE, 2018: 98-109.
[2]   TONG J, ZHOU J, LIU L, et al Scanning 3D full human bodies using kinects[J]. IEEE Transactions on Visualization and Computer Graphics, 2012, 18 (4): 643- 650
doi: 10.1109/TVCG.2012.56
[3]   CHEN G, LI J, WANG B, et al Reconstructing 3D human models with a kinect[J]. Computer Animation and Virtual Worlds, 2016, 27 (1): 72- 85
doi: 10.1002/cav.1632
[4]   CHEN G, LI J, ZENG J, et al Optimizing human model reconstruction from RGB-D image based on skin detection[J]. Virtual Reality, 2016, 20 (3): 159- 172
doi: 10.1007/s10055-016-0291-y
[5]   WEISS A, HIRSHBERG D, BLACL M J. Home 3D body scans from noisy image and range data [C]// International Conference on Computer Vision. Barcelona: IEEE, 2011: 1951-1958.
[6]   WANG C C L Parameterization and parametric design of mannequins[J]. Computer-Aided Design, 2005, 37 (1): 83- 98
doi: 10.1016/j.cad.2004.05.001
[7]   BEAK S Y, LEE K Parametric human body shape modeling framework for human-centered product design[J]. Computer-Aided Design, 2012, 44 (1): 56- 67
doi: 10.1016/j.cad.2010.12.006
[8]   HUANG J, KWOK T H, ZHOU C Parametric design for human body modeling by wireframe-assisted deep learning[J]. Computer-Aided Design, 2019, 108: 19- 29
doi: 10.1016/j.cad.2018.10.004
[9]   ANGUELOV D, SRINIVASAN P, KOLLER D, et al SCAPE: shape completion and animation of people[J]. ACM Transactions on Graphics, 2005, 24 (3): 408- 416
doi: 10.1145/1073204.1073207
[10]   LOPER M, MAHMOOD N, ROMERO J, et al SMPL: a skinned multi-person linear model[J]. ACM Transactions on Graphics, 2015, 34 (6): 248
[11]   POPA A I, ZANFIR M, SMINCHISESCU C. Deep multitask architecture for integrated 2d and 3d human sensing [C]//Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 6289-6298.
[12]   PAVLAKOS G, ZHU L, ZHOU X, et al. Learning to estimate 3D human pose and shape from a single color image [C]// Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 459-468.
[13]   JI Z, QI X, WANG Y, et al Human body shape reconstruction from binary silhouette images[J]. Computer Aided Geometric Design, 2019, 71: 231- 243
doi: 10.1016/j.cagd.2019.04.019
[14]   KANAZAWA A, BLACK M J, JACOBS D W, et al. End-to-end recovery of human shape and pose [C]// Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7122-7131.
[15]   GUAN P, WEISS A, BALAN A O, et al. Estimating human shape and pose from a single image [C]// International Conference on Computer Vision. Florida: IEEE, 2009: 1381-1388.
[16]   OMRAN M, LASSNER C, PONS-MOLL G, et al. Neural body fitting: unifying deep learning and model based human pose and shape estimation [C]// International Conference on 3D Vision. Verona: IEEE, 2018: 484-494.
[17]   JOHNSON S, EVERINGHAM M. Clustered pose and nonlinear appearance models for human pose estimation [C]// British Machine Vision Conference. Aberystwyth: BMVA, 2010: 5.
[18]   IONESCU C, PAPAVA D, OLARU V, et al Human3.6m: large scale datasets and predictive methods for 3d human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36 (7): 1325- 1339
[19]   LASSNER C, ROMERO J, KIEFEL M, et al. Unite the people: closing the loop between 3d and 2d human representations [C]// Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017: 6050-6059.
[20]   PISHCHULIN L, WUHRER S, HELTEN T, et al Building statistical shape space for 3d human modeling[J]. Patten Recognition, 2017, 67: 276- 286
doi: 10.1016/j.patcog.2017.02.018
[21]   LI J, LU G Customizing 3D garments based on volumetric deformation[J]. Computers in Industry, 2011, 62 (7): 693- 707
doi: 10.1016/j.compind.2011.04.002
[22]   FREIFELD O, BLACK M J. Lie bodies: a manifold representation of 3D human shape [C]// European Conference on Computer Vision. Berlin: Springer, 2012: 1-14.
[23]   FLETCHER P T, LU C, JOSHI S. Statistics of shape via principal geodesic analysis on lie groups [C]// IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Wisconsin: IEEE, 2003: 95-101.
[24]   LECUN Y, BOTTOU L, BENGIO Y, et al Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86 (11): 2278- 2324
doi: 10.1109/5.726791
[25]   HINTON G E, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors [J]. Computer Science, 2012, 3(4): 212-223.
[1] Jia-hui XU,Jing-chang WANG,Ling CHEN,Yong WU. Surface water quality prediction model based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 601-607.
[2] Hong-li WANG,Bin GUO,Si-cong LIU,Jia-qi LIU,Yun-gang WU,Zhi-wen YU. End context-adaptative deep sensing model with edge-end collaboration[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 626-638.
[3] Teng ZHANG,Xin-long JIANG,Yi-qiang CHEN,Qian CHEN,Tao-mian MI,Piu CHAN. Wrist attitude-based Parkinson's disease ON/OFF state assessment after medication[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 639-647.
[4] Li-feng XU,Hai-fan HUANG,Wei-long DING,Yu-lei FAN. Detection of small fruit target based on improved DenseNet[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(2): 377-385.
[5] Yi-peng HUANG,Ji-su HU,Xu-sheng QIAN,Zhi-yong ZHOU,Wen-lu ZHAO,Qi MA,Jun-kang SHEN,Ya-kang DAI. SE-Mask-RCNN: segmentation method for prostate cancer on multi-parametric MRI[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(1): 203-212.
[6] Pu ZHENG,Hong-yang BAI,Wei LI,Hong-wei GUO. Small target detection algorithm in complex background[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1777-1784.
[7] Qiao-hong CHEN,YI CHEN,Wen-shu Li,Yu-bo JIA. Clothing image classification based on multi-scale SE-Xception[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1727-1735.
[8] Deng-wen ZHOU,Jin-yue TIAN,Lu-yao MA,Xiu-xiu SUN. Lightweight image semantic segmentation based on multi-level feature cascaded network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(8): 1516-1524.
[9] Tao MING,Dan WANG,Ji-chang GUO,Qiang LI. Breast cancer histopathological image classification using multi-scale channel squeeze-and-excitation model[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(7): 1289-1297.
[10] Xu YAN,Xiao-liang FAN,Chuan-pan ZHENG,Yu ZANG,Cheng WANG,Ming CHENG,Long-biao CHEN. Urban traffic flow prediction algorithm based on graph convolutional neural networks[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(6): 1147-1155.
[11] Zhou-fei WANG,Wei-na YUAN. Channel estimation and detection method for multicarrier system based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(4): 732-738.
[12] Bing YANG,Wen-bo MO,Jin-liang YAO. 3D palmprint recognition by using local features and deep learning[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 540-545.
[13] Yan-jia HONG,Tie-bao MENG,Hao-jiang LI,Li-zhi LIU,Li LI,Shuo-yu XU,Sheng-wen GUO. Deep segmentation method of tumor boundaries from MR images of patients with nasopharyngeal carcinoma using multi-modality and multi-dimension fusion[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 566-573.
[14] Zi-yu JIA,You-fang LIN,Hong-jun ZHANG,Jing WANG. Sleep stage classification model based ondeep convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1899-1905.
[15] Wan-liang WANG,Xiao-han YANG,Yan-wei ZHAO,Nan GAO,Chuang LV,Zhao-juan ZHANG. Image enhancement algorithm with convolutional auto-encoder network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(9): 1728-1740.