Proceedings of the 2016 International Conference on Advanced Electronic Science and Technology (AEST 2016)

Improved YARN resource scheduling algorithm based on network load sensing

Authors
Xuyang Ding, Xiaohui Liu, Rui Zhang, Shan Tang, Linqin Sun, Ying Xie
Corresponding Author
Xuyang Ding
Available Online November 2016.
DOI
10.2991/aest-16.2016.114How to use a DOI?
Keywords
multitask scheduling; YARN; task flow graph; greedy strategy; network resource.
Abstract

Confronted with multitask scheduling, YARN may conduct a precise calculation and allocation towards every single task's computing and memory resources,however it failed to consider the impact on task fulfillment and cluster resources utilization. In this paper, we propose a new algorithm to allocate the data interconnected tasks to the same node or to another closely-located low network load node. This new algorithm firstly builds a task flow graph for the task requests of an application, and then assigns tasks more properly using the greedy strategy, based on the state of network resource. In this way, it significantly improves the management performance of YARN in the case of network congestion or instability. Besides, it consumesless execution time, makes higher utilization of CPU and requires lower network load.

Copyright
© 2016, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 2016 International Conference on Advanced Electronic Science and Technology (AEST 2016)
Series
Advances in Intelligent Systems Research
Publication Date
November 2016
ISBN
10.2991/aest-16.2016.114
ISSN
1951-6851
DOI
10.2991/aest-16.2016.114How to use a DOI?
Copyright
© 2016, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Xuyang Ding
AU  - Xiaohui Liu
AU  - Rui Zhang
AU  - Shan Tang
AU  - Linqin Sun
AU  - Ying Xie
PY  - 2016/11
DA  - 2016/11
TI  - Improved YARN resource scheduling algorithm based on network load sensing
BT  - Proceedings of the 2016 International Conference on Advanced Electronic Science and Technology (AEST 2016)
PB  - Atlantis Press
SP  - 852
EP  - 859
SN  - 1951-6851
UR  - https://doi.org/10.2991/aest-16.2016.114
DO  - 10.2991/aest-16.2016.114
ID  - Ding2016/11
ER  -