The Improvement of Network Performance by Using the Technique of PF-RING Zero-copy to Optimize Spark Streaming
- DOI
- 10.2991/cnct-16.2017.9How to use a DOI?
- Keywords
- Zero copy, PF_RING, Spark Streaming, Adaptive
- Abstract
Regarding the problem of lower utilization rate of broadband and data throughput in the real-time computations when Spark Streaming receiving and treating a large number of TCP packet, we proposed a technique of effective zero-copy of IO kernel and optimiazation of user protocol stack. By comparing and analyzing the monitoring data, we can determine that the reason of low utilization rate of network under special circumstances is the large expanse of kernel system which induced by the big data flow frame dealing with a large number of network IO interruptions in unit time. We propose that the realization of protocol stack can be stripped from the kernel of operating system by combining PF_RING, this can both reduce the expanse of system management and avoid the additional data movement. The results demonstrate that comparing with the present open-source version of Spark-1.4.0, we can effectively reduce the time delay, decrease the jitter and improved the utilization rate of the Worker nodes by using the technique of PF-RING zero-copy to optimize Spark Streaming. And this technique can also meet the real-time computation's need when receiving and treating a large number of TCP packets.
- Copyright
- © 2017, the Authors. Published by Atlantis Press.
- Open Access
- This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Cite this article
TY - CONF AU - Feng LIU AU - Zhan-rong LI AU - Yang GAO AU - Sheng-si LUO PY - 2016/12 DA - 2016/12 TI - The Improvement of Network Performance by Using the Technique of PF-RING Zero-copy to Optimize Spark Streaming BT - Proceedings of the International Conference on Computer Networks and Communication Technology (CNCT 2016) PB - Atlantis Press SP - 62 EP - 68 SN - 2352-538X UR - https://doi.org/10.2991/cnct-16.2017.9 DO - 10.2991/cnct-16.2017.9 ID - LIU2016/12 ER -