PCJ Java library as a solution to integrate HPC, Big Data and Artificial Intelligence workloads

Journal of Big Data

Marek Nowicki¹  Piotr Bała²  Łukasz Górski² 

[1] Faculty of Mathematics and Computer Science, Nicolaus Copernicus University in Toruń, ul. Chopina 12/18, 87-100, Toruń, Poland;Interdisciplinary Centre for Mathematical and Computational Modeling, University of Warsaw, ul. Tyniecka 15/17, 02-630, Warsaw, Poland;

关键词: Parallel computing;    Java;    Partitioned Global Address Space;    PCJ;    HPC;    Big Data;    Artifical Intelligence;   

DOI  :  10.1186/s40537-021-00454-6

来源: Springer

【 摘 要 】

With the development of peta- and exascale size computational systems there is growing interest in running Big Data and Artificial Intelligence (AI) applications on them. Big Data and AI applications are implemented in Java, Scala, Python and other languages that are not widely used in High-Performance Computing (HPC) which is still dominated by C and Fortran. Moreover, they are based on dedicated environments such as Hadoop or Spark which are difficult to integrate with the traditional HPC management systems. We have developed the Parallel Computing in Java (PCJ) library, a tool for scalable high-performance computing and Big Data processing in Java. In this paper, we present the basic functionality of the PCJ library with examples of highly scalable applications running on the large resources. The performance results are presented for different classes of applications including traditional computational intensive (HPC) workloads (e.g. stencil), as well as communication-intensive algorithms such as Fast Fourier Transform (FFT). We present implementation details and performance results for Big Data type processing running on petascale size systems. The examples of large scale AI workloads parallelized using PCJ are presented.

【 授权许可】

CC BY