Processing of the WLCG monitoring data using NoSQL

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Andreeva, J.^1 ; Beche, A.^1 ; Belov, S.^2 ; Dzhunov, I.^1 ; Kadochnikov, I.^2 ; Karavakis, E.^1 ; Saiz, P.^1 ; Schovancova, J.^3 ; Tuckett, D.^1

CERN, European Organization for Nuclear Research, Switzerland^1

Joint Inst. for Nuclear Research, Russia^2

Brookhaven National Laboratory, United States^3

关键词: Commodity hardware;    Computing activity;    Large-scale monitoring;    LHC computing grids;    LHC experiments;    Monitoring system;    Nosql database;    Performance and reliabilities;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032048/pdf
DOI  :  10.1088/1742-6596/513/3/032048

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The Worldwide LHC Computing Grid (WLCG) today includes more than 150 computing centres where more than 2 million jobs are being executed daily and petabytes of data are transferred between sites. Monitoring the computing activities of the LHC experiments, over such a huge heterogeneous infrastructure, is extremely demanding in terms of computation, performance and reliability. Furthermore, the generated monitoring flow is constantly increasing, which represents another challenge for the monitoring systems. While existing solutions are traditionally based on Oracle for data storage and processing, recent developments evaluate NoSQL for processing large-scale monitoring datasets. NoSQL databases are getting increasingly popular for processing datasets at the terabyte and petabyte scale using commodity hardware. In this contribution, the integration of NoSQL data processing in the Experiment Dashboard framework is described along with first experiences of using this technology for monitoring the LHC computing activities.