Please wait a minute...
浙江大学学报(工学版)
服务计算     
面向大数据试验场应用的资源优化分配
白如帆, 雷建坤, 张亮
复旦大学 计算机科学技术学院 上海数据科学重点实验室,上海 201203
Towards resource allocation optimization for big data test field application
BAI Ru-fan, LEI Jian-kun, ZHANG Liang
College of Computer Science Technology, Shanghai Data Science Key Laboratory, Fudan University, Shanghai 201203, China
 全文: PDF(1048 KB)   HTML
摘要:

针对多个用户申请有限的资源的情况,提出两段式优化资源分配方法,在满足用户需求的同时确保资源的公平有效分配.根据以往的执行日志自动确定单类应用相对于资源投入量的饱和点.根据占优资源公平分享原则,为每类应用动态确定可投入运行的实例数量,实现全局最优化配置,在平台层面体现资源共享、杜绝欺诈、公平最优以及帕累托均衡等特性.选择Clik+公用的基准测试程序集,以Docker容器作为应用运行环境验证应用的普遍性,实验结果表明:两段式优化资源分配方法提升了资源利用率,且在多用户同时申请资源时优化了资源配置.模拟并检验了两阶段优化资源分配算法的可行性及有效性.

Abstract: A two-phase optimization resource allocation solution was proposed to ensure fair and efficient allocation and meet the multiple users’ requirement, who applied for limited resources. First, the trade-off point in term of performance and resources for each type of application was determined automatically according to prevous logs. Then, according to the Dominant Resource Fairness principle, the number of instances that could be put into operation was automatically determined for each class of application to realize implement global optimized allocation. Therefore, characterastics like sharing incentive, strategy-proofness, envy-freeness and Pareto equilibrium on the system level were reflected. The solution’s generality was validated using the Clik+ benchmark package with Docker containers as operating environment. Results demonstrate that the two-phase optimization resource allocation solution can improve resource utilization, which can also optimize resource configuration when several users apply for resources simultaneously. Moreover, the feasibility and effectiveness of the proposed program was simulated and validated.
出版日期: 2017-06-11
CLC:  TU 111  
基金资助:

国家自然科学基金资助项目(60873115); 教育部-中国移动科研基金资助项目(MCM20123011); 上海市科技发展基金资助项目(13dz2260200, 13511504300).

通讯作者: 张亮,男,教授.     E-mail: lzhang@fudan.edu.cn
作者简介: 白如帆(1995—),男,硕士生,从事服务计算研究. ORCID: 0000-0001-5535-5114. E-mail: rfbai@fudan.edu.cn
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  

引用本文:

白如帆, 雷建坤, 张亮. 面向大数据试验场应用的资源优化分配[J]. 浙江大学学报(工学版), 10.3785/j.issn.1008-973X.2017.06.021.

BAI Ru-fan, LEI Jian-kun, ZHANG Liang. Towards resource allocation optimization for big data test field application. JOURNAL OF ZHEJIANG UNIVERSITY (ENGINEERING SCIENCE), 10.3785/j.issn.1008-973X.2017.06.021.

参考文献(References):
[1] 李吉萍.复旦大学稳步推进大数据试验场建设[EB/OL].[2016-08-29].http:∥news.fudan.edu.cn/2016/0829/42127.html.
[2] 上海数据科学重点实验室.发展农业大数据产业的关键[R].[2016-07-09].http:∥wenku.baidu.com/view/4de92d88783e0912a3162aa9.html.
[3] 赵颖. Hadoop 环境下的动态资源管理研究与实现[D]. 上海交通大学, 2015.
ZHAO Ying. Research and implementation of dynamic resource management on Hadoop [D]. Shanghai: JiaoTong University, 2015.
[4] GHODSI A, ZAHARIA M, HINDMAN B, et al. Dominant resource fairness: fair allocation of multiple resource types [C] ∥ NSDI 2011.Boston: [s. n.], 2011: 24.
[5] KANBUR R. Pareto’s revenge [M]. New York: Cornell University, 2005: 2-8.
[6] TOMOIAGA B, CHINDRIS M, SUMPER A, et al. Pareto optimal reconfiguration of power distribution systems using a genetic algorithm based on NSGAII [J]. Energies, 2013, 6 (3): 1439-1455.
[7] VIJAY K. How well do we know Pareto optimality? [J]. The Journal of Economic Education, 1991, 22(2): 172-178.
[8] Cilk+的基准测试程序集[CP/DK]. Charles E. Leiserson: MIT CSAIL Supertech Research Group, 2010.
[9] BALALAIE A, HEVDARNOORI A, JAMSHIDI P. Microservices architecture enables DevOps: migration to a cloud-native architecture [J]. IEEE Software,2016, 33(3): 42-52.
[10] ALPERS S, BECKER C, OBERWEIS A, et al. Microservice based tool support for business process modelling [C] ∥ Enterprise Distributed Object Computing Workshop. Adelaide: IEEE, 2015: 71-78.
[11] NADAREISHVILI I, MITRA R, MCLARTY M, et al. Microservice architecture: aligning principles, practices, and culture [M]. O’Reilly Media, Inc, 2016. http:∥shop.oreilly.com/product/0636920050308.do
[12] RICHARDSON C. Introduction to microservices [EB/OL]. [2015-05-19].https:∥www.nginx.com/blog/introduction-to-microservices.
[13] DRAGONI N, GIALLORENZO S, LAFUENTE A L, et al. Microservices: yesterday, today, and tomorrow [J/OL]. arXiv preprint arXiv: 1606.04036. 2016: 1-17.
https:∥arxiv.org/abs/1606.04036.
[14] LE B, JEAN Y. Rate adaptation, congestion control and fairness: a tutorial [EB/OL]. (2012-08-23) [2016--11-21]. http:∥ica1www.epfl.ch/PS_files/LEB3132.pdf.
[15] JOE W, SEN S, LAN T, et al. Multiresource allocation: fairness-efficiency tradeoffs in a unifying framework [J]. IEEE/ACM Transactions on Networking, 2013, 21(6):
1785-1798.
[16] PARKES D, PROCACCIA A, SHAH N. Beyond dominant resource fairness: extensions, limitations, and indivisibilities [J]. ACM Transactions on Economics and Computation, 2015, 3(1): 1-18.
[17] BRIAN T. Finding the best trade-off point on a curve [EB/OL]. (2010-01-07)[2016-11-21]. http:∥stackoverflow.com/questions/2018178/finding-the-best-trade-off-point-on-a-curve.
[18] SUH C, MO J. Resource allocation for multicast services in multicarrier wireless communications [J]. IEEE Transactions on Wireless Communications, 2008, 7(1): 27-31.
[19] HAHNE E. Round-robin scheduling for max-min fairness in data networks [J]. IEEE Journal on Selected Areas in Communications, 1991, 9(7): 1024-1039.
[20] LAN T, KAO D, CHIANG M, et al. An axiomatic theory of fairness in network resource allocation [M]. [S.l]: IEEE, 2010: 1-17.
[1] 王亮,谢海建,吴家葳,陈云敏,丁昊. 填埋场成层衬垫污染物运移参数等效模型[J]. 浙江大学学报(工学版), 2021, 55(3): 519-529.
[2] 蔡亮,周泓岑,白恒,才振功,尹可挺,贝毅君. 基于多层BiLSTM和改进粒子群算法的应用负载预测方法[J]. 浙江大学学报(工学版), 2020, 54(12): 2414-2422.
[3] 陆江,王登辉,赵康,刘诗韵. 自保温建筑不同末端间歇供暖的实测效果分析[J]. 浙江大学学报(工学版), 2020, 54(11): 2092-2099.
[4] 张岩,郭斌,王倩茹,张靖,於志文. SeqRec:基于长期偏好和即时兴趣的序列推荐模型[J]. 浙江大学学报(工学版), 2020, 54(6): 1177-1184.
[5] 冷伍明,张期树,徐方,冷慧康,聂如松,杨秀航. 预应力路堤附加围压场与围压增强效应[J]. 浙江大学学报(工学版), 2020, 54(5): 858-869.
[6] 何绍衡,夏唐代,于丙琪,丁智,高敏,单华峰. 温度效应对钙质砂体积应变和固结特性的影响[J]. 浙江大学学报(工学版), 2020, 54(2): 221-232.
[7] 袁海辉,葛一敏,甘春标. 不确定性扰动下双足机器人动态步行的自适应鲁棒控制[J]. 浙江大学学报(工学版), 2019, 53(11): 2049-2057.
[8] 杜丽,梅标,朱伟东,柯臻铮. 采用模糊区间分析的柔性航空结构件装配偏差预测[J]. 浙江大学学报(工学版), 2019, 53(9): 1647-1655.
[9] 葛坚,王登辉,赵康. 夏热冬冷地区供暖房间冷风渗透影响因素[J]. 浙江大学学报(工学版), 2019, 53(7): 1415-1422.
[10] 何绍衡,夏唐代,李连祥,于丙琪,刘泽勇. 地下水渗流对悬挂式止水帷幕基坑变形影响[J]. 浙江大学学报(工学版), 2019, 53(4): 713-723.
[11] 黄钰期,梅盼,陈晓济,邓长水. 基于CFD分析的电动汽车电池包加热方法[J]. 浙江大学学报(工学版), 2019, 53(2): 207-213.
[12] 阮方, 钱晓倩, 钱匡亮, 余亚超, 施水华. 夏热冬冷地区居住建筑外墙外保温的反节能现象[J]. 浙江大学学报(工学版), 2016, 50(12): 2343-2349.
[13] 董凯, 赖俊英, 钱晓倩, 詹树林, 阮方. 夏热冬冷地区居住建筑水平式外遮阳节能效果[J]. 浙江大学学报(工学版), 2016, 50(8): 1431-1437.
[14] 阮方,钱晓倩,朱耀台,吴敏莉. 分室间歇用能对墙体内外保温节能效果的影响[J]. 浙江大学学报(工学版), 2016, 50(1): 1-7.
[15] 易思阳,金虹庆,范利武,徐旭,俞自涛,葛坚. 多孔建筑材料水蒸气扩散系数的瞬态测试方法[J]. 浙江大学学报(工学版), 2016, 50(1): 16-20.