The future of PanDA in ATLAS distributed computing

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

De, K.^1 ; Klimentov, A.^2 ; Maeno, T.^2 ; Nilsson, P.^2 ; Oleynik, D.^1 ; Panitkin, S.^2 ; Petrosyan, A.^3 ; Schovancova, J.^1 ; Vaniachine, A.^4 ; Wenaus, T.^2

University of Texas, Arlington

TX, United States^1

Brookhaven National Laboratory, NY, United States^2

Joint Institute for Nuclear Research, Dubna, Russia^3

Argonne National Laboratory, IL, United States^4

关键词: ATLAS experiment;    Computing infrastructures;    Computing workloads;    Distributed analysis;    Heterogeneous resources;    Information sources;    Large Hadron collider LHC;    Workload management;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/6/062035/pdf
DOI  :  10.1088/1742-6596/664/6/062035

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Experiments at the Large Hadron Collider (LHC) face unprecedented computing challenges. Heterogeneous resources are distributed worldwide at hundreds of sites, thousands of physicists analyse the data remotely, the volume of processed data is beyond the exabyte scale, while data processing requires more than a few billion hours of computing usage per year. The PanDA (Production and Distributed Analysis) system was developed to meet the scale and complexity of LHC distributed computing for the ATLAS experiment. In the process, the old batch job paradigm of locally managed computing in HEP was discarded in favour of a far more automated, flexible and scalable model. The success of PanDA in ATLAS is leading to widespread adoption and testing by other experiments. PanDA is the first exascale workload management system in HEP, already operating at more than a million computing jobs per day, and processing over an exabyte of data in 2013. There are many new challenges that PanDA will face in the near future, in addition to new challenges of scale, heterogeneity and increasing user base. PanDA will need to handle rapidly changing computing infrastructure, will require factorization of code for easier deployment, will need to incorporate additional information sources including network metrics in decision making, be able to control network circuits, handle dynamically sized workload processing, provide improved visualization, and face many other challenges. In this talk we will focus on the new features, planned or recently implemented, that are relevant to the next decade of distributed computing workload management using PanDA.