Improving Performance in Hadoop Using Automatic and Predictive Configuration
- Juan Fang, Hao Sun, Li-Fu Zhou, Xing-Tian Ren, Min Cai
- Corresponding Author
- Juan Fang
Available Online November 2016.
- https://doi.org/10.2991/ceis-16.2016.54How to use a DOI?
- high performance; dynamic; prediction; big data
- MapReduce is an effective programming model for analyzing large-scale data. Hadoop-a distributed processing system is widely used nowadays. Improving the task parallelism can be a key point to improve the MapReduce performance in Hadoop. In this paper, we address the problem in two ways. On the one hand we can run the tasks with some dynamic configurations. On the other hand, considering of the difference of tasktracker we use mathematics method to predict the cups' utilization of tasktracker to assign the task. Experimental results on both ways show we can improve the performance in Hadoop by improving the task parallelism.
- Open Access
- This is an open access article distributed under the CC BY-NC license.
Cite this article
TY - CONF AU - Juan Fang AU - Hao Sun AU - Li-Fu Zhou AU - Xing-Tian Ren AU - Min Cai PY - 2016/11 DA - 2016/11 TI - Improving Performance in Hadoop Using Automatic and Predictive Configuration BT - 2016 International Conference on Computer Engineering and Information Systems PB - Atlantis Press SN - 2352-538X UR - https://doi.org/10.2991/ceis-16.2016.54 DO - https://doi.org/10.2991/ceis-16.2016.54 ID - Fang2016/11 ER -