Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2025, Vol. 59 Issue (11): 2400-2408    DOI: 10.3785/j.issn.1008-973X.2025.11.019
    
Reinforcement learning-based scheduling algorithm for cloud-edge collaborative computing on Kubernetes
Jiawei TANG1(),Tiezheng GUO1,Yingyou WEN1,2,*()
1. School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China
2. Neusoft Group Limited Company, Shenyang 110179, China
Download: HTML     PDF(783KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A reinforcement learning-based cloud-edge collaborative computing resource scheduling algorithm, KNCS, was proposed aiming at the problem of insufficient resource utilization in cloud-edge collaborative computing scenarios due to imbalances in network and computational resources, as well as uncertainties in task types and arrival times. This algorithm achieved shorter transmission time, processing time, and turnaround time by comprehensively considering the state of network resource and computational resource. A unified information transmission platform was designed to aggregate information from computational nodes and various tasks, facilitating the definition of task dependencies, dynamically adjusting subsequent tasks based on the type of running tasks, and providing a more realistic task scheduling scenario. The experimental results show that the performance of the KNCS algorithm surpasses that of the default Kubernetes scheduling algorithm in cloud-edge collaborative computing scenario.

Key words： cloud-edge collaborative computing      internet of thing      task scheduling      reinforcement learning algorithm      distributed computing     
Received: 28 October 2024      Published: 30 October 2025
CLC:  TP 393  
Fund:  国家自然科学基金资助项目（62172084, 92167103）.
Corresponding Authors: Yingyou WEN     E-mail: 2301944@stu.neu.edu.cn;wenyingyou@mail.neu.edu.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Jiawei TANG
Tiezheng GUO
Yingyou WEN
Cite this article:

Jiawei TANG,Tiezheng GUO,Yingyou WEN. Reinforcement learning-based scheduling algorithm for cloud-edge collaborative computing on Kubernetes. Journal of ZheJiang University (Engineering Science), 2025, 59(11): 2400-2408.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2025.11.019     OR     https://www.zjujournals.com/eng/Y2025/V59/I11/2400


基于强化学习的Kubernetes云边协同计算调度算法

针对云边协同计算在网络资源和计算资源不平衡、任务类型和到达时间不确定的场景中存在资源利用不充分的问题，提出基于强化学习的云边协同计算资源调度算法KNCS. 通过综合考虑网络资源和计算资源的状态，该算法实现了更短的传输时间、处理时间和周转时间. 设计统一的信息传输平台，聚合来自计算节点和各个任务的信息，支持任务依赖关系的定义，根据运行任务的类型动态调整后续任务，提供更真实的任务调度场景. 实验结果表明，在云边协同计算场景下，KNCS算法的性能优于默认的Kubernetes调度算法.


关键词： 云边协同计算,  物联网,  任务调度,  强化学习算法,  分布式计算 
Fig.1 Architecture of unified information transceiver platform
Fig.2 Architecture diagram of scheduling system
算法1　KNCS调度算法
输入：随机概率ε、动作空间A
1.　首次运行时初始化经验池$ E $和神经网络参数$ {\theta }_{{\mathrm{policy}}} $
2.　接收调度请求
3.　以ε的概率随机选择动作$ {A}_{i} $
4.　否则选择$ {A}_{i}={\mathrm{argma}}{{\mathrm{x}}}_{{A}_{i}}Q\left({S}_{i},{A}_{i},{\theta }_{{\mathrm{policy}}}\right) $
5.　调度决策J$ \leftarrow {A}_{i} $
6.　更新经验池$ E $
输出：调度决策J
 
算法2　KNCS训练算法
输入：经验池$ E $、衰减系数$ \alpha $
1.　初始化策略网络和目标网络参数$ {\theta }_{{\mathrm{policy}}} $、$ {\theta }_{{\mathrm{target}}} $
2.　for 每一次训练迭代 do
3.　　从 $ E $ 中抽取一批经验
4.　　if $ {N}_{i}==1 $ then
5.　　　将$ \left({S}_{i},{A}_{i},{R}_{i},{S}_{{\mathrm{next}}\left(i\right)}\right) $添加到经验池$ E $
6.　　　$ {Q}_{{\mathrm{target}}}\leftarrow -{T}_{i,{\mathrm{total}}} $
7.　　else
8.　　　
$ {Q}_{{\mathrm{target}}}\leftarrow \left(1-\alpha \right){\mathrm{ma}}{{\mathrm{x}}}_{{A}'}Q\left({S}_{{\mathrm{next}}\left(i\right)},{A}',{\theta }_{{\mathrm{target}}}\right)-\alpha {T}_{i,{\mathrm{total}}} $
9.　　end if
10.　　$ {Q}_{{\mathrm{policy}}}\leftarrow Q\left({S}_{i},{A}_{i},{\theta }_{{\mathrm{policy}}}\right) $
11.　　使用损失函数 $ L={\left({Q}_{{\mathrm{target}}}-{Q}_{{\mathrm{policy}}}\right)}^{2} $ 更新 $ {\theta }_{{\mathrm{policy}}} $
12.　　每 $ N $ 次迭代使用指数移动平均值更新 $ {\theta }_{{\mathrm{target}}} $
13.　end for
输出：迭代后的网络权重$ {\theta }_{\mathrm{p}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{y}} $
 
Fig.3 Schematic diagram of network topology of video processing task nodes
Fig.4 Schematic diagram of network topology of corpus processing task nodes
节点NM/GB
node148
node2-node422
node5-node1012
Tab.1 Configuration of virtualized node
参数数值
训练批量大小128
指数滑动平均系数(奖励衰减系数)0.9
起始随机概率1.0
最小随机概率0.05
学习率10?3
指数滑动平均系数(参数更新)0.6
Tab.2 Configuration of KNCS algorithm parameter
算法Tavg/sTw/sTtrans/sTcom/s
KNCS133.621.56117.3816.24
默认算法187.251.27169.3617.89
Koord175.891.26158.8517.03
Tab.3 Comparison of average performance of algorithm in video processing task
算法Tavg/sTw/sTtrans/sTcom/s
KNCS209.161.61135.7973.38
默认算法316.131.41240.8775.27
Koord270.631.41193.0477.59
Tab.4 Comparison of average performance of algorithm in corpus processing task
Fig.5 Comparison of probability distribution of various indicator in video processing task
Fig.6 Comparison of probability distribution of various indicator in corpus processing task
算法Tavg/s
视频处理任务链语料处理任务链
KNCS380.82513.42
默认算法548.65795.89
Koord503.22688.65
Tab.5 Comparison of average time spent on task chain
[1]   王凌, 吴楚格, 范文慧, 等 边缘计算资源分配与任务调度优化综述[J]. 系统仿真学报, 2021, 33 (3): 509- 520
WANG Ling, WU Chuge, FAN Wenhui, et al A survey of edge computing resource allocation and task scheduling optimization[J]. Journal of System Simulation, 2021, 33 (3): 509- 520
[2]   施巍松, 张星洲, 王一帆, 等 边缘计算: 现状与展望[J]. 计算机研究与发展, 2019, 56 (1): 69- 89
SHI Weisong, ZHANG Xingzhou, WANG Yifan, et al Edge computing: state-of-the-art and future directions[J]. Journal of Computer Research and Development, 2019, 56 (1): 69- 89
doi: 10.7544/issn1000-1239.2019.20180760
[3]   KHAN W Z, AHMED E, HAKAK S, et al Edge computing: a survey[J]. Future Generation Computer Systems, 2019, 97: 219- 235
doi: 10.1016/j.future.2019.02.050
[4]   ERMOLENKO D, KILICHEVA C, MUTHANNA A, et al. Internet of things services orchestration framework based on Kubernetes and edge computing [C]//IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering. Moscow: IEEE, 2021: 12-17.
ERMOLENKO D, KILICHEVA C, MUTHANNA A, et al. Internet of things services orchestration framework based on Kubernetes and edge computing [C]//IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering. Moscow: IEEE, 2021: 12-17.
[5]   VARGHESE B, WANG N, BARBHUIYA S, et al. Challenges and opportunities in edge computing [C]//IEEE International Conference on Smart Cloud. New York: IEEE, 2016: 20-26.
VARGHESE B, WANG N, BARBHUIYA S, et al. Challenges and opportunities in edge computing [C]//IEEE International Conference on Smart Cloud. New York: IEEE, 2016: 20-26.
[6]   SHI W, CAO J, ZHANG Q, et al Edge computing: vision and challenges[J]. IEEE Internet of Things Journal, 2016, 3 (5): 637- 646
[7]   BHARDWAJ A, KRISHNA C R Virtualization in cloud computing: moving from hypervisor to containerization: a survey[J]. Arabian Journal for Science and Engineering, 2021, 46 (9): 8585- 8601
[8]   QIU T, CHI J, ZHOU X, et al Edge computing in industrial internet of things: architecture, advances and challenges[J]. IEEE Communications Surveys and Tutorials, 2020, 22 (4): 2462- 2488
[9]   NING H, LI Y, SHI F, et al Heterogeneous edge computing open platforms and tools for internet of things[J]. Future Generation Computer Systems, 2020, 106: 67- 76
[10]   SATYANARAYANAN M, BAHL P, CACERES R, et al The case for vm-based cloudlets in mobile computing[J]. IEEE Pervasive Computing, 2009, 8 (4): 14- 23
[11]   ZHANG M, CAO J, YANG L, et al. Ents: an edge-native task scheduling system for collaborative edge computing [C]//IEEE/ACM 7th Symposium on Edge Computing. Seattle: IEEE, 2022: 149-161.
[12]   ZHANG M, CAO J, SAHNI Y, et al. Eaas: a service-oriented edge computing framework towards distributed intelligence [C]//IEEE International Conference on Service-Oriented System Engineering. Newark: IEEE, 2022: 165-175.
[13]   HAN R, WEN S, LIU C H, et al. EdgeTuner: fast scheduling algorithm tuning for dynamic edge-cloud workloads and resources [C]//IEEE Conference on Computer Communications. London: IEEE, 2022: 880-889.
[14]   SHAN C, GAO R, HAN Q, et al KCES: a workflow containerization scheduling scheme under cloud-edge collaboration framework[J]. IEEE Internet of Things Journal, 2024, 12 (2): 2026- 2042
[15]   SAHNI Y, CAO J, YANG L Data-aware task allocation for achieving low latency in collaborative edge computing[J]. IEEE Internet of Things Journal, 2018, 6 (2): 3512- 3524
[16]   SHAN C, WANG G, XIA Y, et al. Containerized workflow builder for Kubernetes [C]//IEEE 23rd International Conference on High Performance Computing and Communications; 7th International Conference on Data Science and Systems; 19th International Conference on Smart City; 7th International Conference on Dependability in Sensor, Cloud and Big Data Systems and Application. Haikou: IEEE, 2021: 685-692.
SHAN C, WANG G, XIA Y, et al. Containerized workflow builder for Kubernetes [C]//IEEE 23rd International Conference on High Performance Computing and Communications; 7th International Conference on Data Science and Systems; 19th International Conference on Smart City; 7th International Conference on Dependability in Sensor, Cloud and Big Data Systems and Application. Haikou: IEEE, 2021: 685-692.
[17]   SHAN C, XIA Y, ZHAN Y, et al KubeAdaptor: a docking framework for workflow containerization on Kubernetes[J]. Future Generation Computer Systems, 2023, 148: 584- 599
[18]   BADER J, THAMSEN L, KULAGINA S, et al. Tarema: adaptive resource allocation for scalable scientific workflows in heterogeneous clusters [C]//IEEE International Conference on Big Data. Orlando: IEEE, 2021: 65-75.
[19]   GAREFALAKIS P, KARANASOS K, PIETZUCH P, et al. Medea: scheduling of long running applications in shared production clusters [C]//European Conference on Computer Systems. New York: ACM, 2018: 1-13.
[20]   HAO Y, JIANG Y, CHEN T, et al iTaskOffloading: intelligent task offloading for a cloud-edge collaborative system[J]. IEEE Network, 2019, 33 (5): 82- 88
[21]   GUO K, YANG M, ZHANG Y, et al Joint computation offloading and bandwidth assignment in cloud-assisted edge computing[J]. IEEE Transactions on Cloud Computing, 2019, 10 (1): 451- 460
[22]   YANG L, YANG D, CAO J, et al QoS guaranteed resource allocation for live virtual machine migration in edge clouds[J]. IEEE Access, 2020, 8: 78441- 78451
[23]   TAN B, MA H, MEI Y, et al A cooperative coevolution genetic programming hyper-heuristics approach for on-line resource allocation in container-based clouds[J]. IEEE Transactions on Cloud Computing, 2020, 10 (3): 1500- 1514
[24]   VERMA A, PEDROSA L, KORUPOLU M, et al. Large-scale cluster management at Google with Borg [C]//Proceedings of the 10th European Conference on Computer Systems. New York: ACM, 2015: 1-17.
[25]   XIONG Y, SUN Y, XING L, et al. Extend cloud to edge with Kubeedge [C]//IEEE/ACM Symposium on Edge Computing. Seattle: IEEE, 2018: 373-377.
XIONG Y, SUN Y, XING L, et al. Extend cloud to edge with Kubeedge [C]//IEEE/ACM Symposium on Edge Computing. Seattle: IEEE, 2018: 373-377.
[26]   DUPONT C, GIAFFREDA R, CAPRA L. Edge computing in IoT context: horizontal and vertical Linux container migration [C]//Global Internet of Things Summit. Geneva: IEEE, 2017: 1-4.
[27]   GOETHALS T, DE TURCK F, VOLCKAERT B. Fledge: Kubernetes compatible container orchestration on low-resource edge devices [C]//International Conference on Internet of Vehicles. Cham: Springer, 2019: 174-189.
[28]   LIU N, LI Z, XU J, et al. A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning [C]//IEEE 37th International Conference on Distributed Computing Systems. Atlanta: IEEE, 2017: 372-382.
[29]   MAO H, ALIZADEH M, MENACHE I, et al. Resource management with deep reinforcement learning [C]//Proceedings of the 15th ACM Workshop on Hot Topics in Networks. New York: ACM, 2016: 50-56.
[30]   YI D, ZHOU X, WEN Y, et al Efficient compute-intensive job allocation in data centers via deep reinforcement learning[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31 (6): 1474- 1485
[31]   MAO H, SCHWARZKOPF M, VENKATAKRISHNAN S B, et al. Learning scheduling algorithms for data processing clusters [C]//Proceedings of the ACM Special Interest Group on Data Communication. New York: ACM, 2019: 270-288.
MAO H, SCHWARZKOPF M, VENKATAKRISHNAN S B, et al. Learning scheduling algorithms for data processing clusters [C]//Proceedings of the ACM Special Interest Group on Data Communication. New York: ACM, 2019: 270-288.
[32]   邝祝芳, 陈清林, 李林峰, 等 基于深度强化学习的多用户边缘计算任务卸载调度与资源分配算法[J]. 计算机学报, 2022, 45 (4): 812- 824
KUANG Zhufang, CHEN Qinglin, LI Linfeng, et al Multi-user edge computing task offloading scheduling and resource allocation based on deep reinforcement learning[J]. Chinese Journal of Computers, 2022, 45 (4): 812- 824
doi: 10.11897/SP.J.1016.2022.00812
[33]   周陈静, 骆淑云 基于深度强化学习的实时视频边缘卸载策略[J]. 智能计算机与应用, 2024, 14 (8): 32- 39
ZHOU Chenjing, LUO Shuyun Computation offloading decision in video edge computing based on deep reinforcement learning[J]. Intelligent Computer and Applications, 2024, 14 (8): 32- 39
[34]   张斐斐, 葛季栋, 李忠金, 等 边缘计算中协作计算卸载与动态任务调度[J]. 软件学报, 2023, 34 (12): 5737- 5756
ZHANG Feifei, GE Jidong, LI Zhongjin, et al Cooperative computation offloading and dynamic task scheduling in edge computing[J]. Journal of Software, 2023, 34 (12): 5737- 5756
[1] Yi HU,Mengsheng CUI,Xiyang ZHANG,Yanqing ZHAO. Task scheduling optimization for automated guided vehicle based on directed acyclic graph modeling[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(8): 1680-1688.
[2] Jiaojun LI,Tao YU,Jihua ZHOU,Fan YANG,Tao ZHAO,Tianshu WU,Zilin MA. Resource allocation strategy for cognitive industrial internet of things in dynamic uncertain scenarios[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(5): 960-966.
[3] Yanfang WANG,Wei WANG,Yunquan DONG. Study of timeliness and distortion performance for real-time decision making in IoT[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(4): 664-673.
[4] WU Chao, LIU Yuan-an, WU Fan, FAN Wen-hao, TANG Bi-hua. Efficient data gathering scheme in mobility-constrained internet of things with graph theory[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(8): 1444-1451.
[5] REN Zhi-yuan, HOU Xiang-wang, GUO Kai, ZHANG Hai-lin, CHEN Chen. Distributed satellite cloud-fog network and strategy of latency and power consumption[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(8): 1474-1481.
[6] SHENG Nian-zu, LI Fang, LI Xiao-feng, ZHAO He, ZHOU Tong. Data capitalization method based on blockchain smart contract for Internet of Things[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(11): 2150-2158.
[7] LI Jian-li, DING Ding, LI Tao. Multi-objective hybrid cloud task scheduling using twice clustering[J]. Journal of ZheJiang University (Engineering Science), 2017, 51(6): 1233-1241.
[8] LIU Duan-yang , Xie Jian-ping, CAO Yan-long. Research on divisible load scheduling algorithm based on energy model[J]. Journal of ZheJiang University (Engineering Science), 2013, 47(9): 1547-1553.
[9] XU Ke-Jun, HU Wen-Yao, CHEN Ji-Zhong, et al. Task scheduling model and algorithm based on dual-Vdd dynamic reconfigurable FPGA[J]. Journal of ZheJiang University (Engineering Science), 2010, 44(2): 300-304.
[10] XING Mei, LIU Feng, GU Xiong-Li, et al. Study on implementation of real-time operating system that supports multi-threading processor[J]. Journal of ZheJiang University (Engineering Science), 2009, 43(7): 1177-1181.