Monitoring of large-scale federated data storage: XRootD and beyond

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Andreeva, J.^1 ; Beche, A.^1 ; Belov, S.^2 ; Arias, D. Diguez^1 ; Giordano, D.^1 ; Oleynik, D.^2 ; Petrosyan, A.^2 ; Saiz, P.^1 ; Tadel, M.^3 ; Tuckett, D.^1 ; Vukotic, I.^4

CERN (European Organization for Nuclear Research), Switzerland^1

JINR (Joint Institute for Nuclear Research), Russia^2

UCSD (University of California, San Diego), United States^3

U. Chicago (University of Chicago), United States^4

关键词: Computing model;    Data distribution policies;    Data federation;    Hierarchical data;    LHC experiments;    Monitoring architecture;    Monitoring frameworks;    Resource optimization;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032004/pdf
DOI  :  10.1088/1742-6596/513/3/032004

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The computing models of the LHC experiments are gradually moving from hierarchical data models with centrally managed data pre-placement towards federated storage which provides seamless access to data files independently of their location and dramatically improve recovery due to fail-over mechanisms. Construction of the data federations and understanding the impact of the new approach to data management on user analysis requires complete and detailed monitoring. Monitoring functionality should cover the status of all components of the federated storage, measuring data traffic and data access performance, as well as being able to detect any kind of inefficiencies and to provide hints for resource optimization and effective data distribution policy. Data mining of the collected monitoring data provides a deep insight into new usage patterns. In the WLCG context, there are several federations currently based on the XRootD technology. This paper will focus on monitoring for the ATLAS and CMS XRootD federations implemented in the Experiment Dashboard monitoring framework. Both federations consist of many dozens of sites accessed by many hundreds of clients and they continue to grow in size. Handling of the monitoring flow generated by these systems has to be well optimized in order to achieve the required performance. Furthermore, this paper demonstrates the XRootD monitoring architecture is sufficiently generic to be easily adapted for other technologies, such as HTTP/WebDAV dynamic federations.