Proceedings of the 2016 International Conference on Computer Engineering and Information Systems

Improving Performance in Hadoop Using Automatic and Predictive Configuration

Authors
Juan Fang, Hao Sun, Li-Fu Zhou, Xing-Tian Ren, Min Cai
Corresponding Author
Juan Fang
Available Online November 2016.
DOI
https://doi.org/10.2991/ceis-16.2016.54How to use a DOI?
Keywords
high performance; dynamic; prediction; big data
Abstract
MapReduce is an effective programming model for analyzing large-scale data. Hadoop-a distributed processing system is widely used nowadays. Improving the task parallelism can be a key point to improve the MapReduce performance in Hadoop. In this paper, we address the problem in two ways. On the one hand we can run the tasks with some dynamic configurations. On the other hand, considering of the difference of tasktracker we use mathematics method to predict the cups' utilization of tasktracker to assign the task. Experimental results on both ways show we can improve the performance in Hadoop by improving the task parallelism.
Open Access
This is an open access article distributed under the CC BY-NC license.

Download article (PDF)

Proceedings
2016 International Conference on Computer Engineering and Information Systems
Part of series
Advances in Computer Science Research
Publication Date
November 2016
ISBN
978-94-6252-283-1
DOI
https://doi.org/10.2991/ceis-16.2016.54How to use a DOI?
Open Access
This is an open access article distributed under the CC BY-NC license.

Cite this article

TY  - CONF
AU  - Juan Fang
AU  - Hao Sun
AU  - Li-Fu Zhou
AU  - Xing-Tian Ren
AU  - Min Cai
PY  - 2016/11
DA  - 2016/11
TI  - Improving Performance in Hadoop Using Automatic and Predictive Configuration
BT  - 2016 International Conference on Computer Engineering and Information Systems
PB  - Atlantis Press
UR  - https://doi.org/10.2991/ceis-16.2016.54
DO  - https://doi.org/10.2991/ceis-16.2016.54
ID  - Fang2016/11
ER  -