Please wait a minute...
浙江大学学报(工学版)  2019, Vol. 53 Issue (7): 1315-1322    DOI: 10.3785/j.issn.1008-973X.2019.07.010
自动化技术、计算机技术     
室内环境中人穿携物品归属关系自主学习框架
吴皓1(),李文静1,田国会1,*(),陈兆伟1,杨勇2
1. 山东大学 控制科学与工程学院,山东 济南 250061
2. 济南皓源自控系统工程有限公司,山东 济南 250061
Self-learning framework for attribution relationship of people carrying objects under family environment
Hao WU1(),Wen-jing LI1,Guo-hui TIAN1,*(),Zhao-wei CHEN1,Yong YANG2
1. School of Control Science and Engineering, Shandong University, Jinan 250061, China
2. Jinan Haoyuan Automatic Control System Engineering Limited Company, Jinan 250061, China
 全文: PDF(1331 KB)   HTML
摘要:

为了满足机器人个性化服务需求,使机器人可以根据不同的服务对象来选择专属物品进行任务的推理和规划,需要服务机器人具备自主获取人穿携物品与人的归属关系的能力. 针对家庭环境下人穿携物品与人的归属关系获取问题,提出人穿携物品归属关系自主学习框架. 采用基于物品检测模型SSD与人体姿态估计模型OpenPose相结合的人穿携物品检测定位方法,实现人穿携物品检测. 利用基于迁移学习的卷积神经网络提取物品特征,通过后端分类器完成物品实例属性识别,使用人脸检测与识别模型MTCNN完成服务对象身份识别. 通过归属关系自主学习策略,完成归属关系的自主学习. 实验结果表明,提出的人穿携物品归属关系自主学习框架能够准确、高效地完成归属关系的学习,有效排除环境干扰因素对归属关系学习的影响,具有良好的准确性和鲁棒性.

关键词: 服务机器人个性化服务深度学习穿携物品物品归属关系    
Abstract:

It is necessary for service robots to have the ability to independently obtain the attribution relationship between people and their carrying objects in order to satisfy the requirements of robot personalized service and enable robots to select exclusive objects to perform inference and planning according to different service individual. A self-learning framework for the attribution relationship between people and their carrying objects was proposed aiming at the problem of the attribution relationship between people carrying objects and people in the family environment. The method of detecting and locating people carrying objects was used based on the object detection model SSD and the human posture estimation model OpenPose in order to realize the detection of human carrying objects. The face detection and recognition model MTCNN were used to complete the service individual identification by extracting the objects features by convolutional neural network based on migration learning and using the backend classifier to complete the object instance attribute identification. The self-learning of the attribution relationship was completed through the self-learning strategy. The experimental results show that the proposed self-learning framework for attribution relationship of people carrying objects can accurately and efficiently complete the learning of attribution relationship, effectively eliminating the influence of environmental interference factors on attribution learning. The proposed framework has good accuracy and robustness.

Key words: service robot    personalized service    deep learning    carrying object    attribution relationship of objects
收稿日期: 2018-10-31 出版日期: 2019-06-25
CLC:  TP 242  
通讯作者: 田国会     E-mail: wh911@sdu.edu.cn;g.h.tian@sdu.edu.cn
作者简介: 吴皓(1972—),女,副教授,从事智能机器人的研究. orcid.org/0000-0001-6993-8863. E-mail: wh911@sdu.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
RSS
作者相关文章  
吴皓
李文静
田国会
陈兆伟
杨勇

引用本文:

吴皓,李文静,田国会,陈兆伟,杨勇. 室内环境中人穿携物品归属关系自主学习框架[J]. 浙江大学学报(工学版), 2019, 53(7): 1315-1322.

Hao WU,Wen-jing LI,Guo-hui TIAN,Zhao-wei CHEN,Yong YANG. Self-learning framework for attribution relationship of people carrying objects under family environment. Journal of ZheJiang University (Engineering Science), 2019, 53(7): 1315-1322.

链接本文:

http://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2019.07.010        http://www.zjujournals.com/eng/CN/Y2019/V53/I7/1315

图 1  物品检测与定位流程图
图 2  物品实例识别流程
图 3  人脸识别流程图
图 4  人脸检测效果图
图 5  短期记忆矩阵存储结构与实例
图 6  长期记忆矩阵存储结构与实例
图 7  归属关系选择算法流程
图 8  TurTleBot移动机器人平台
图 9  物品检测定位实验
图 10  物品实例识别实验
图 11  物品实例识别准确率
图 12  人脸识别混淆矩阵
图 13  短期记忆矩阵
图 14  长期归属关系记忆矩阵
图 15  归属物品查询结果
1 HSIEH J W, CHENG J C, CHEN L C, et al Handheld object detection and its related event analysis using ratio histogram and mixture of HMMs[J]. Journal of Visual Communication and Image Representation, 2014, 25 (6): 1399- 1415
doi: 10.1016/j.jvcir.2014.05.009
2 RIVERA-RUBIO J, IDREES S, ALEXIOU I, et al. A dataset for hand-held object recognition [C] // IEEE International Conference on Image Processing. Paris: IEEE, 2015: 5881-5885.
3 LV X, JIANG S Q, HERRANZ L, et al RGB-D hand-held object recognition based on heterogeneous feature fusion[J]. Computer Science and Technology, 2015, 30 (2): 340- 352
doi: 10.1007/s11390-015-1527-0
4 LI X, JIANG S Q, LV X, et al. Learning to recognize hand-held objects from scratch [C] // Advances in Multimedia Information Processing. Berlin: Springer, 2016: 527–539.
5 YAMAGUCHI K, KIAPOUR M H, ORTIZ L E, et al. Parsing clothing in fashion photographs [C] // Computer Vision and Pattern Recognition. Phode Island: IEEE, 2012: 3570-3577.
6 LIANG X D, XU C Y, SHEN X H, et al. Human parsing with contextualized convolutional neural network [C] // IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1386-1394.
7 GONG K, LIANG X D, ZHANG D Y, et al. Look into person: self-supervised structure-sensitive learning and a new benchmark for human parsing [C] // IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 932-940.
8 CHEN X J, MOTTAGHI R, LIU X B, et al. Detect what you can: detecting and representing objects using holistic models and body parts [C] // Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 1979-1986.
9 LI J S, ZHAO J, WEI Y C, et al. Towards real world human parsing: multiple-human parsing in the wild [C] // Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017.
10 LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector [C] // European Conference on Computer Vision. Berlin: Springer, 2016: 21-37.
11 LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection [C] // Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2999-3007.
12 IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift [C] // Proceedings of the 32nd International Conference on Machine Learning. Coimbra: ACM, 2015: 448-456.
13 SARAFIANOS N, BOTEANU B, IONESCU B, et al 3D human pose estimation: a review of the literature and analysis of covariates[J]. Computer Vision and Image Understanding, 2016, 152: 1- 20
doi: 10.1016/j.cviu.2016.09.002
14 CAO Z, SIMON T, WEI S E, et al. Realtime multi-person 2D pose estimation using part affinity fields [C] // Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1302-1310.
15 CHENG M M. Saliency and similarity detection for image scene analysis [D]. Tianjin: NanKai University, 2012: 33-56.
16 ZHAO Z Q, HUANG D S, SUN B Y Human face recognition based on multi-features using neural networks committee[J]. Pattern Recognition Letters, 2004, 25 (12): 1351- 1358
doi: 10.1016/j.patrec.2004.05.008
17 GEETHA K P, VADIVELU S S, SINGH N A. Human face recognition using neural networks [C] // Radio Science Conference. Cairo: IEEE, 2012: 260-263.
18 YANG S, LUO P, CHEN C L, et al. WIDER FACE: a face detection benchmark [C] // Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 5525-5533.
19 LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild [C] // Proceedings of the IEEE International Conference on Computer Vision. Boston: IEEE, 2015: 3730-3738.
20 WANG M, DENG W. Deep face recognition: a survey [C] // Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 471-478.
21 ZHANG K, ZHANG Z, LI Z, et al Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23 (10): 1499- 1503
doi: 10.1109/LSP.2016.2603342
[1] 杨婧,耿辰,王海林,纪建松,戴亚康. 基于DenseNet的低分辨CT影像肺腺癌组织学亚型分类[J]. 浙江大学学报(工学版), 2019, 53(6): 1164-1170.
[2] 吕艳,张萌,姜吴昊,倪益华,钱小鸿. 采用卷积神经网络的老年人跌倒检测系统设计[J]. 浙江大学学报(工学版), 2019, 53(6): 1130-1138.
[3] 李泚泚, 田国会, 张梦洋, 张营. 基于本体的物品属性类人认知及推理[J]. 浙江大学学报(工学版), 2018, 52(7): 1231-1238.
[4] 袁公萍, 汤一平, 韩旺明, 陈麒. 基于深度卷积神经网络的车型识别方法[J]. 浙江大学学报(工学版), 2018, 52(4): 694-702.
[5] 沈延斌, 陈岭, 郭浩东, 陈根才. 基于深度学习的放置方式和位置无关运动识别[J]. 浙江大学学报(工学版), 2016, 50(6): 1141-1148.