Preparing the Gaudi framework and the DIRAC WMS for multicore job submission

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Rauschmayr, N.^1 ; Streit, A.^2

CERN-European Organization for Nuclear Research, Geneva, Switzerland^1

Steinbuch Centre for Computing, Karlsruhe Institute of Technology, Karlsruhe, Germany^2

关键词: Memory consumption;    Memory footprint;    New technologies;    Parallel prototypes;    Parallelizations;    Scheduling strategies;    Software frameworks;    Workload management;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/5/052029/pdf
DOI  :  10.1088/1742-6596/513/5/052029

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

HEP applications need to adapt to the continuously increasing number of cores on modern CPUs. This must be done at different levels: the software must support parallelization, and the scheduling has to differ between multicore and singlecore jobs. The LHCb software framework (GAUDI) provides a parallel prototype (GaudiMP), based on the multiprocessing approach. It allows a reduction of the overall memory footprint and a coordinated access to data via separated reader and writer processes. A comparison between the parallel prototype and multiple independent Gaudi jobs in respect of CPU time and memory consumption will be shown. Furthermore, speedup must be predicted in order to find the limit beyond which the parallel prototype (GaudiMP) does not bring further scaling. This number must be known as it indicates the point, where new technologies must be introduced into the software framework. In order to reach further improvements in the overall throughput, scheduling strategies for mixing parallel jobs can be applied. It allows overcoming limitations in the speedup of the parallel prototype. Those changes require modifications at the level of the Workload Management System (DIRAC).