Distributed Data Collection for the ATLAS EventIndex

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Sánchez, J.^1 ; Casaní, A Fernández^1 ; González De La Hoz, S.^1

Instituto de Fisica Corpuscular, Universitat de Valencia, CSIC, Valencia, Spain^1

关键词: Distributed data;    Messaging system;    Processing stage;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/4/042046/pdf
DOI  :  10.1088/1742-6596/664/4/042046

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The ATLAS EventIndex contains records of all events processed by ATLAS, in all processing stages. These records include the references to the files containing each event (the GUID of the file) and the internal pointer to each event in the file. This information is collected by all jobs that run at Tier-0 or on the Grid and process ATLAS events. Each job produces a snippet of information for each permanent output file. This information is packed and transferred to a central broker at CERN using an ActiveMQ messaging system, and then is unpacked, sorted and reformatted in order to be stored and catalogued into a central Hadoop server. This contribution describes in detail the Producer/Consumer architecture to convey this information from the running jobs through the messaging system to the Hadoop server.