Task Management in the New ATLAS Production System

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

De, K.^1 ; Golubkov, D.^2 ; Klimentov, A.^3 ; Potekhin, M.^3 ; Vaniachine, A.^4

University of Texas at Arlington, Arlington

TX, United States^1

IHEP, Protvino, Russia^2

Brookhaven National Laboratory, Upton

NY, United States^3

Argonne National Laboratory, Argonne

IL, United States^4

关键词: ATLAS experiment;    Database engine;    Design parameters;    Production level;    Production system;    Production workloads;    Task management;    Workload management;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032078/pdf
DOI  :  10.1088/1742-6596/513/3/032078

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

This document describes the design of the new Production System of the ATLAS experiment at the LHC [1]. The Production System is the top level workflow manager which translates physicists' needs for production level processing and analysis into actual workflows executed across over a hundred Grid sites used globally by ATLAS. As the production workload increased in volume and complexity in recent years (the ATLAS production tasks count is above one million, with each task containing hundreds or thousands of jobs) there is a need to upgrade the Production System to meet the challenging requirements of the next LHC run while minimizing the operating costs. In the new design, the main subsystems are the Database Engine for Tasks (DEFT) and the Job Execution and Definition Interface (JEDI). Based on users' requests, DEFT manages inter-dependent groups of tasks (Meta-Tasks) and generates corresponding data processing workflows. The JEDI component then dynamically translates the task definitions from DEFT into actual workload jobs executed in the PanDA Workload Management System [2]. We present the requirements, design parameters, basics of the object model and concrete solutions utilized in building the new Production System and its components.