A Fine-Grained Task Monitoring Mechanism in Spark Platform
- Cheng Chen, Fei Liu, Guangrui Li, Xiang Chen, Ying Hou
- Corresponding Author
- Cheng Chen
Available Online June 2017.
- https://doi.org/10.2991/ammee-17.2017.75How to use a DOI?
- Spark, In-memory Computing, Fine-grained, Task monitoring.
- The existing coarse-grained monitoring mechanism in Spark has much difficulty in analyzing and locating the bottleneck during the execution of tasks. Aiming to solve this problem, we divide the execution process of Spark tasks into several subphases, and propose a fine-grained subphases-based monitoring mechanism. The fine-grained monitoring mechanism can help users to understand the detail execution status of Spark tasks and be beneficial to analyze and locate the performance bottleneck in Spark.
- Open Access
- This is an open access article distributed under the CC BY-NC license.
Cite this article
TY - CONF AU - Cheng Chen AU - Fei Liu AU - Guangrui Li AU - Xiang Chen AU - Ying Hou PY - 2017/06 DA - 2017/06 TI - A Fine-Grained Task Monitoring Mechanism in Spark Platform BT - Advances in Materials, Machinery, Electrical Engineering (AMMEE 2017) PB - Atlantis Press UR - https://doi.org/10.2991/ammee-17.2017.75 DO - https://doi.org/10.2991/ammee-17.2017.75 ID - Chen2017/06 ER -