Proceedings of the International Conference on Computer Networks and Communication Technology (CNCT 2016)

The Improvement of Network Performance by Using the Technique of PF-RING Zero-copy to Optimize Spark Streaming

Authors
Feng LIU, Zhan-rong LI, Yang GAO, Sheng-si LUO
Corresponding Author
Feng LIU
Available Online December 2016.
DOI
https://doi.org/10.2991/cnct-16.2017.9How to use a DOI?
Keywords
Zero copy, PF_RING, Spark Streaming, Adaptive
Abstract

Regarding the problem of lower utilization rate of broadband and data throughput in the real-time computations when Spark Streaming receiving and treating a large number of TCP packet, we proposed a technique of effective zero-copy of IO kernel and optimiazation of user protocol stack. By comparing and analyzing the monitoring data, we can determine that the reason of low utilization rate of network under special circumstances is the large expanse of kernel system which induced by the big data flow frame dealing with a large number of network IO interruptions in unit time. We propose that the realization of protocol stack can be stripped from the kernel of operating system by combining PF_RING, this can both reduce the expanse of system management and avoid the additional data movement. The results demonstrate that comparing with the present open-source version of Spark-1.4.0, we can effectively reduce the time delay, decrease the jitter and improved the utilization rate of the Worker nodes by using the technique of PF-RING zero-copy to optimize Spark Streaming. And this technique can also meet the real-time computation's need when receiving and treating a large number of TCP packets.

Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

Volume Title
Proceedings of the International Conference on Computer Networks and Communication Technology (CNCT 2016)
Series
Advances in Computer Science Research
Publication Date
December 2016
ISBN
978-94-6252-301-2
ISSN
2352-538X
DOI
https://doi.org/10.2991/cnct-16.2017.9How to use a DOI?
Copyright
© 2017, the Authors. Published by Atlantis Press.
Open Access
This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

TY  - CONF
AU  - Feng LIU
AU  - Zhan-rong LI
AU  - Yang GAO
AU  - Sheng-si LUO
PY  - 2016/12
DA  - 2016/12
TI  - The Improvement of Network Performance by Using the Technique of PF-RING Zero-copy to Optimize Spark Streaming
BT  - Proceedings of the International Conference on Computer Networks and Communication Technology (CNCT 2016)
PB  - Atlantis Press
SP  - 62
EP  - 68
SN  - 2352-538X
UR  - https://doi.org/10.2991/cnct-16.2017.9
DO  - https://doi.org/10.2991/cnct-16.2017.9
ID  - LIU2016/12
ER  -