Proceedings of the 2018 Joint International Advanced Engineering and Technology Research Conference (JIAET 2018)

<Previous Article In Volume
Next Article In Volume>

Multi-link query optimization based on heuristic search in data stream

Authors
Zhiming Chen, Fengzhen Wang, Boyang Liu
Corresponding Author
Zhiming Chen
Available Online March 2018.
DOI
10.2991/jiaet-18.2018.4How to use a DOI?
Keywords
heuristic search; data stream; undirected weighted graph; connection tree
Abstract

In the stream processing, the connection operation is one of the very important basic operations, and it plays an important role in real-time data query and mining. According to the existing data stream connection technology and combining with the characteristics of coarse granularity processing unit of Sparkstreaming platform, this paper designs a multiple connection query optimization algorithm based on heuristic search. The algorithm first transforms the query into an undirected weighted graph, constructs the connection tree through the heuristic function to optimize the connection order, and combines the sliding window technology to introduce the cache mechanism and reduce the computational amount. Due to the characteristic that during the stream processing the data is coming continuously and constantly, the function evaluation value is periodically updated and the connection tree is rebuilt dynamically, in order to ensure the continuous and efficient connection operation. The experimental part uses the distributed Message Queue Kafka to generate multiple data streams, and uses Sparkstreaming as a consumer to connect operations and verify the feasibility of the algorithm.

Copyright
© 2018, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

<Previous Article In Volume
Next Article In Volume>
Volume Title
Proceedings of the 2018 Joint International Advanced Engineering and Technology Research Conference (JIAET 2018)
Series
Advances in Engineering Research
Publication Date
March 2018
ISBN
10.2991/jiaet-18.2018.4
ISSN
2352-5401
DOI
10.2991/jiaet-18.2018.4How to use a DOI?
Copyright
© 2018, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

risenwbib
TY  - CONF
AU  - Zhiming Chen
AU  - Fengzhen Wang
AU  - Boyang Liu
PY  - 2018/03
DA  - 2018/03
TI  - Multi-link query optimization based on heuristic search in data stream
BT  - Proceedings of the 2018 Joint International Advanced Engineering and Technology Research Conference (JIAET 2018)
PB  - Atlantis Press
SP  - 19
EP  - 23
SN  - 2352-5401
UR  - https://doi.org/10.2991/jiaet-18.2018.4
DO  - 10.2991/jiaet-18.2018.4
ID  - Chen2018/03
ER  -