| Service Computing |
|
|
|
|
| Towards resource allocation optimization for big data test field application |
| BAI Ru-fan, LEI Jian-kun, ZHANG Liang |
| College of Computer Science Technology, Shanghai Data Science Key Laboratory, Fudan University, Shanghai 201203, China |
|
|
|
Abstract A two-phase optimization resource allocation solution was proposed to ensure fair and efficient allocation and meet the multiple users’ requirement, who applied for limited resources. First, the trade-off point in term of performance and resources for each type of application was determined automatically according to prevous logs. Then, according to the Dominant Resource Fairness principle, the number of instances that could be put into operation was automatically determined for each class of application to realize implement global optimized allocation. Therefore, characterastics like sharing incentive, strategy-proofness, envy-freeness and Pareto equilibrium on the system level were reflected. The solution’s generality was validated using the Clik+ benchmark package with Docker containers as operating environment. Results demonstrate that the two-phase optimization resource allocation solution can improve resource utilization, which can also optimize resource configuration when several users apply for resources simultaneously. Moreover, the feasibility and effectiveness of the proposed program was simulated and validated．
|
|
Published: 11 June 2017
|
|
|
|
Cite this article:
BAI Ru-fan, LEI Jian-kun, ZHANG Liang. Towards resource allocation optimization for big data test field application. JOURNAL OF ZHEJIANG UNIVERSITY (ENGINEERING SCIENCE), 2017, 51(6): 1225-1232.
|
面向大数据试验场应用的资源优化分配
针对多个用户申请有限的资源的情况,提出两段式优化资源分配方法，在满足用户需求的同时确保资源的公平有效分配.根据以往的执行日志自动确定单类应用相对于资源投入量的饱和点.根据占优资源公平分享原则,为每类应用动态确定可投入运行的实例数量,实现全局最优化配置,在平台层面体现资源共享、杜绝欺诈、公平最优以及帕累托均衡等特性.选择Clik+公用的基准测试程序集,以Docker容器作为应用运行环境验证应用的普遍性,实验结果表明：两段式优化资源分配方法提升了资源利用率,且在多用户同时申请资源时优化了资源配置.模拟并检验了两阶段优化资源分配算法的可行性及有效性.
|
|
| 参考文献(References)：
［1］ 李吉萍.复旦大学稳步推进大数据试验场建设［EB/OL］.［2016-08-29］.http:∥news.fudan.edu.cn/2016/0829/42127.html．
［2］ 上海数据科学重点实验室.发展农业大数据产业的关键［R］.［2016-07-09］.http:∥wenku.baidu.com/view/4de92d88783e0912a3162aa9.html.
［3］ 赵颖. Hadoop 环境下的动态资源管理研究与实现［D］. 上海交通大学, 2015．
ZHAO Ying. Research and implementation of dynamic resource management on Hadoop ［D］. Shanghai: JiaoTong University, 2015.
［4］ GHODSI A, ZAHARIA M, HINDMAN B, et al. Dominant resource fairness: fair allocation of multiple resource types ［C］ ∥ NSDI 2011.Boston: ［s. n.］, 2011: 24．
［5］ KANBUR R. Pareto’s revenge ［M］. New York: Cornell University, 2005: 2-8．
［6］ TOMOIAGA B, CHINDRIS M, SUMPER A, et al. Pareto optimal reconfiguration of power distribution systems using a genetic algorithm based on NSGAII ［J］. Energies, 2013, 6 (3): 1439-1455．
［7］ VIJAY K. How well do we know Pareto optimality？ ［J］. The Journal of Economic Education, 1991, 22(2): 172-178．
［8］ Cilk+的基准测试程序集［CP/DK］. Charles E. Leiserson: MIT CSAIL Supertech Research Group, 2010．
［9］ BALALAIE A, HEVDARNOORI A, JAMSHIDI P. Microservices architecture enables DevOps: migration to a cloud-native architecture ［J］. IEEE Software,2016, 33(3): 42-52.
［10］ ALPERS S, BECKER C, OBERWEIS A, et al. Microservice based tool support for business process modelling ［C］ ∥ Enterprise Distributed Object Computing Workshop. Adelaide: IEEE, 2015: 71-78．
［11］ NADAREISHVILI I, MITRA R, MCLARTY M, et al. Microservice architecture: aligning principles, practices, and culture ［M］. O’Reilly Media, Inc, 2016. http:∥shop.oreilly.com/product/0636920050308.do
［12］ RICHARDSON C. Introduction to microservices ［EB/OL］. ［2015-05-19］.https:∥www.nginx.com/blog/introduction-to-microservices．
［13］ DRAGONI N, GIALLORENZO S, LAFUENTE A L, et al. Microservices: yesterday, today, and tomorrow ［J/OL］. arXiv preprint arXiv: 1606.04036. 2016: 1-17.
https:∥arxiv.org/abs/1606.04036.
［14］ LE B, JEAN Y. Rate adaptation, congestion control and fairness: a tutorial ［EB/OL］. (2012-08-23) ［2016--11-21］. http:∥ica1www.epfl.ch/PS_files/LEB3132.pdf．
［15］ JOE W, SEN S, LAN T, et al. Multiresource allocation: fairness-efficiency tradeoffs in a unifying framework ［J］. IEEE/ACM Transactions on Networking, 2013, 21(6):
1785-1798．
［16］ PARKES D, PROCACCIA A, SHAH N. Beyond dominant resource fairness: extensions, limitations, and indivisibilities ［J］. ACM Transactions on Economics and Computation, 2015, 3(1): 1-18．
［17］ BRIAN T. Finding the best trade-off point on a curve ［EB/OL］. (2010-01-07)［2016-11-21］. http:∥stackoverflow.com/questions/2018178/finding-the-best-trade-off-point-on-a-curve．
［18］ SUH C, MO J. Resource allocation for multicast services in multicarrier wireless communications ［J］. IEEE Transactions on Wireless Communications, 2008, 7(1): 27-31．
［19］ HAHNE E. Round-robin scheduling for max-min fairness in data networks ［J］. IEEE Journal on Selected Areas in Communications, 1991, 9(7): 1024-1039．
［20］ LAN T, KAO D, CHIANG M, et al. An axiomatic theory of fairness in network resource allocation ［M］. ［S.l］: IEEE, 2010: 1-17． |
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
