Proceedings of the 2016 4th International Conference on Machinery, Materials and Information Technology Applications

A kind of Prefetching Data Way to Hadoop MapReduce Environments

Authors
Hui Xia, Peng Wu
Corresponding Author
Hui Xia
Available Online January 2017.
DOI
10.2991/icmmita-16.2016.234How to use a DOI?
Keywords
Hadoop; MapReduce; Prefetching ;distributed computing.
Abstract

In MapReduce environment, problem of data redundancy,a large number of tasks to be processing and mass data storage come up, in order to solve these problems, we put forward the way of data prefetching , preprocessing of hid the remote data access latency, by adjusting the allocation of resources to Reduce the business, we put forward the method of than before, caused by the Reduce tasks of remote data access performance problems caused by time delay and resource competition system, the hidden by prefetching data the method Reduce task of remote data access latency, and Reduce task control through the resource allocation, to Reduce resource competition caused the Reduce task, the experiment results show that with the default Hadoop graphs and Hadoop Online Prototype (HOP), compared with the method the system performance can be improved by more than 10%.

Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 2016 4th International Conference on Machinery, Materials and Information Technology Applications
Series
Advances in Computer Science Research
Publication Date
January 2017
ISBN
978-94-6252-285-5
ISSN
2352-538X
DOI
10.2991/icmmita-16.2016.234How to use a DOI?
Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Hui Xia
AU  - Peng Wu
PY  - 2017/01
DA  - 2017/01
TI  - A kind of Prefetching Data Way to Hadoop MapReduce Environments
BT  - Proceedings of the 2016 4th International Conference on Machinery, Materials and Information Technology Applications
PB  - Atlantis Press
SN  - 2352-538X
UR  - https://doi.org/10.2991/icmmita-16.2016.234
DO  - 10.2991/icmmita-16.2016.234
ID  - Xia2017/01
ER  -