Resource control in ATLAS distributed data management: Rucio Accounting and Quotas

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Barisits, M.^1,2 ; Serfon, C.^1 ; Garonne, V.^3 ; Lassnig, M.^1 ; Beermann, T.^1 ; Vigne, R.^1

CERN, Geneva, Switzerland^1

Vienna University of Technology, Vienna, Austria^2

University of Oslo, Oslo, Norway^3

关键词: ATLAS experiment;    Data management system;    Distributed data managements;    Production system;    Quota systems;    Resource control;    Resource management;    Storage elements;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/6/062002/pdf
DOI  :  10.1088/1742-6596/664/6/062002

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The ATLAS Distributed Data Management system manages more than 160PB of physics data across more than 130 sites globally. Rucio, the next generation Distributed Data Management system of the ATLAS experiment, replaced DQ2 in December 2014 and will manage the experiment's data throughout Run 2 of the LHC and beyond. The previous data management system pursued a rather simplistic approach for resource management, but with the increased data volume and more dynamic handling of data workflows required by the experiment, a more elaborate approach is needed. Rucio was delivered with an initial quota system, but during the first months of operation it turned out to not fully satisfy the collaboration's resource management needs. We consequently introduce a new concept of declaring quota policies (limits) for accounts in Rucio. This new quota concept is based on accounts and RSE (Rucio storage element) expressions, which allows the definition of hierarchical quotas in a dynamic way. This concept enables the operators of the data management system to implement very specific policies for users, physics groups and production systems while, at the same time, lowering the operational burden. This contribution describes the concept, architecture and workflow of the system and includes an evaluation measuring the performance of the system.