Data federation strategies for ATLAS using XRootD

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Gardner, Robert^1 ; Campana, Simone^4 ; Duckeck, Guenter^2 ; Elmsheuser, Johannes^2 ; Hanushevsky, Andrew^3 ; Hönig, Friedrich G.^2 ; Iven, Jan^4 ; Legger, Federica^2 ; Vukotic, Ilija^1 ; Yang, Wei^3

University of Chicago, Chicago

IL, United States^1

Ludwig-Maximilians-Universität München, Germany^2

Stanford Linear Accelerator Center, Stanford

CA, United States^3

CERN, Geneva, Switzerland^4

关键词: Architecture-based;    Computing resource;    Data management services;    Distributed storage resources;    Integration services;    Monitoring frameworks;    Performance factors;    Production services;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/4/042049/pdf
DOI  :  10.1088/1742-6596/513/4/042049

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job-brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.