Improved YARN resource scheduling algorithm based on network load sensing
Xuyang Ding, Xiaohui Liu, Rui Zhang, Shan Tang, Linqin Sun, Ying Xie
Available Online November 2016.
- https://doi.org/10.2991/aest-16.2016.114How to use a DOI?
- multitask scheduling; YARN; task flow graph; greedy strategy; network resource.
- Confronted with multitask scheduling, YARN may conduct a precise calculation and allocation towards every single task's computing and memory resources,however it failed to consider the impact on task fulfillment and cluster resources utilization. In this paper, we propose a new algorithm to allocate the data interconnected tasks to the same node or to another closely-located low network load node. This new algorithm firstly builds a task flow graph for the task requests of an application, and then assigns tasks more properly using the greedy strategy, based on the state of network resource. In this way, it significantly improves the management performance of YARN in the case of network congestion or instability. Besides, it consumesless execution time, makes higher utilization of CPU and requires lower network load.
- Open Access
- This is an open access article distributed under the CC BY-NC license.
Cite this article
TY - CONF AU - Xuyang Ding AU - Xiaohui Liu AU - Rui Zhang AU - Shan Tang AU - Linqin Sun AU - Ying Xie PY - 2016/11 DA - 2016/11 TI - Improved YARN resource scheduling algorithm based on network load sensing BT - 2016 International Conference on Advanced Electronic Science and Technology (AEST 2016) PB - Atlantis Press SN - 1951-6851 UR - https://doi.org/10.2991/aest-16.2016.114 DO - https://doi.org/10.2991/aest-16.2016.114 ID - Ding2016/11 ER -