Embedding GPU Computations in Hadoop
- DOI
- 10.2991/ijndc.2014.2.4.2How to use a DOI?
- Keywords
- Hadoop, MapReduce, GPU, CUDA
- Abstract
As the size of high performance applications increases, four major challenges including heterogeneity, programmability, fault resilience, and energy efficiency have arisen in the underlying distributed systems. To tackle with all of them without sacrificing performance, traditional approaches in resource utilization, task scheduling and programming paradigm should be reconsidered. While Hadoop has handled data-intensive applications well in Clouds, GPU has demonstrated its acceleration effectiveness for computation-intensive ones. This paper addresses the approaches for Hadoop to exploiting both CPU and GPU resources effectively to handle aforementioned challenges. Hadoop schedules MapReduce’s Map and Reduce functions across multiple different computing nodes through Java, whereas CUDA code helps accelerate local computations further on attached GPUs. All available heterogeneous computational power will be utilized. MapReduce in Hadoop eases the programming task by hiding communication and scheduling details. Hadoop Distributed File System will help achieve data-level fault resilience. GPU’s energy efficiency characteristics help reduce the power consumption of the whole system. To utilize GPU in Hadoop, four approaches including Jcuda, JNI, Hadoop Streaming, and Hadoop Pipes, have been accomplished and analyzed. Experimental results have demonstrated and compared their effectiveness.
- Copyright
- © 2017, the Authors. Published by Atlantis Press.
- Open Access
- This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Cite this article
TY - JOUR AU - Jie Zhu AU - Hai Jiang AU - Juanjuan Li AU - Erikson Hardesty AU - Kuan-Ching Li AU - Zhongwen Li PY - 2014 DA - 2014/10/31 TI - Embedding GPU Computations in Hadoop JO - International Journal of Networked and Distributed Computing SP - 211 EP - 220 VL - 2 IS - 4 SN - 2211-7946 UR - https://doi.org/10.2991/ijndc.2014.2.4.2 DO - 10.2991/ijndc.2014.2.4.2 ID - Zhu2014 ER -