Proceedings of the 2015 3rd International Conference on Machinery, Materials and Information Technology Applications

Self-learning dynamic adjustment scheduling algorithm based on Hadoop

Authors
Fucong Li, Zhuyu Li, Guohui Chen, Xiangxin Li
Corresponding Author
Fucong Li
Available Online November 2015.
DOI
10.2991/icmmita-15.2015.62How to use a DOI?
Keywords
Hadoop; Scheduling algorithm; Speculative execution; Slow tasks
Abstract

Speculative execution is the key technology to improve the execution efficiency of Hadoop cluster. In the large-scale cluster reasonable Speculative execution can effectively reduce the execution time for the job. At present, there is still a big problem in the detecting of the Hadoop's Speculative execution .Therefore this paper proposes a self-learning dynamic adjustment scheduling algorithm. The algorithm firstly study the impact of the historical information of the task execution, dynamically adjusts the time proportions of each stage of Reduce tasks And taking into account the tasks of the different types of load effect on the detection of slow tasks .Found out the real impact of the job execution time of the slow tasks to avoid performing unnecessary backup tasks. Through the experiment, the average execution time of SLDA scheduling is compared with LATE, and the average execution time of the job is significantly reduced.

Copyright
© 2015, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 2015 3rd International Conference on Machinery, Materials and Information Technology Applications
Series
Advances in Computer Science Research
Publication Date
November 2015
ISBN
978-94-6252-120-9
ISSN
2352-538X
DOI
10.2991/icmmita-15.2015.62How to use a DOI?
Copyright
© 2015, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Fucong Li
AU  - Zhuyu Li
AU  - Guohui Chen
AU  - Xiangxin Li
PY  - 2015/11
DA  - 2015/11
TI  - Self-learning dynamic adjustment scheduling algorithm based on Hadoop
BT  - Proceedings of the 2015 3rd International Conference on Machinery, Materials and Information Technology Applications
PB  - Atlantis Press
SP  - 314
EP  - 318
SN  - 2352-538X
UR  - https://doi.org/10.2991/icmmita-15.2015.62
DO  - 10.2991/icmmita-15.2015.62
ID  - Li2015/11
ER  -