Research on Optimization of Big data Storage Structure in Distributed System

DOI

10.2991/icadme-17.2017.63How to use a DOI?

Keywords

Big data; distributed system; row column store.

Abstract

In a distributed system, the storage structure of the data directly affects the storage efficiency and processing performance of big data. In the row storage structure, the data from the local read, loading speed, but the compression efficiency is low and there is data redundancy; in the column storage structure, the data compression efficiency is high, but the data cross-node access increased network transmission consumption The Aiming at the shortcomings of the row storage structure and the column storage structure, a kind of storage method combined with rows and columns is proposed to improve the data storage structure. The experimental results show that the improved data storage structure is slightly lower than the row storage in the loading speed. In the data compression, the efficiency of the parallel storage and the column storage is high. The combined storage structure not only avoids the extra disk I / O overhead, but also reduces the unnecessary storage of the network, which greatly improves the storage efficiency and processing performance of the distributed system for big data.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Zheng-Wu Lu
PY  - 2017/07
DA  - 2017/07
TI  - Research on Optimization of Big data Storage Structure in Distributed System
BT  - Proceedings of the 2017 7th International Conference on Advanced Design and Manufacturing Engineering (ICADME 2017)
PB  - Atlantis Press
SP  - 328
EP  - 333
SN  - 2352-5401
UR  - https://doi.org/10.2991/icadme-17.2017.63
DO  - 10.2991/icadme-17.2017.63
ID  - Lu2017/07
ER  -