Proceedings of the 2017 2nd International Conference on Materials Science, Machinery and Energy Engineering (MSMEE 2017)

Huge Data Analysis and Processing Platform based on Hadoop

Authors
Yuanbin Li, Rong Chen
Corresponding Author
Yuanbin Li
Available Online May 2017.
DOI
10.2991/msmee-17.2017.56How to use a DOI?
Keywords
Huge data analysis, Hadoop, HDFS, MapReduce
Abstract

SGC puts forward the major initiatives of the construction of strong smart grid and three integrates and five big systems, requiring the informationlization further improves to a level on the basis of the inheritance for fruit and deepening application, having even higher requirements especially for huge information data processing, data search grid. In response to meet the needs of the smart grid to the huge amounts of data processing, this article puts forward building dynamic scalable huge amounts of data processing platform on the basis of Hadoop in the virtualization resources management platform through in-depth analysis and research of the two core key technology, HDFS and MapReduce and gives its technical architecture, implementation plan and case analysis. Compared with traditional Hadoop distributed parallel computing system based on the physical machine deployment, to build Hadoop virtual server templates through virtualization platform not only can quickly complete the Hadoop deployment of distributed parallel computing system and can effectively make use of computing resources.

Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the 2017 2nd International Conference on Materials Science, Machinery and Energy Engineering (MSMEE 2017)
Series
Advances in Engineering Research
Publication Date
May 2017
ISBN
10.2991/msmee-17.2017.56
ISSN
2352-5401
DOI
10.2991/msmee-17.2017.56How to use a DOI?
Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Yuanbin Li
AU  - Rong Chen
PY  - 2017/05
DA  - 2017/05
TI  - Huge Data Analysis and Processing Platform based on Hadoop
BT  - Proceedings of the 2017 2nd International Conference on Materials Science, Machinery and Energy Engineering (MSMEE 2017)
PB  - Atlantis Press
SP  - 267
EP  - 271
SN  - 2352-5401
UR  - https://doi.org/10.2991/msmee-17.2017.56
DO  - 10.2991/msmee-17.2017.56
ID  - Li2017/05
ER  -