The ATLAS Event Service: A new approach to event processing

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Calafiura, P.^1 ; De, K.^2 ; Guan, W.^3 ; Maeno, T.^4 ; Nilsson, P.^4 ; Oleynik, D.^2 ; Panitkin, S.^4 ; Tsulaia, V.^1 ; Van Gemmeren, P.^5 ; Wenaus, T.^4

Lawrence Berkeley National Laboratory, 1 Cyclotron Rd, Berkeley

CA

94720, United States^1

University of Texas at Arlington, 701 S Nedderman Dr, Arlington

TX

76019, United States^2

University of Wisconsin - Madison, Madison

WI

53706, United States^3

Brookhaven National Laboratory, Upton

NY

11973, United States^4

Argonne National Laboratory, 9700 South Cass Avenue, Lemont

IL

60439, United States^5

关键词: Distributed processing;    Event Processing;    Input and outputs;    ITS applications;    Processing resources;    Tools and technologies;    U.S. Department of Energy;    Volunteer computing;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/6/062065/pdf
DOI  :  10.1088/1742-6596/664/6/062065

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The ATLAS Event Service (ES) implements a new fine grained approach to HEP event processing, designed to be agile and efficient in exploiting transient, short-lived resources such as HPC hole-filling, spot market commercial clouds, and volunteer computing. Input and output control and data flows, bookkeeping, monitoring, and data storage are all managed at the event level in an implementation capable of supporting ATLAS-scale distributed processing throughputs (about 4M CPU-hours/day). Input data flows utilize remote data repositories with no data locality or pre-staging requirements, minimizing the use of costly storage in favor of strongly leveraging powerful networks. Object stores provide a highly scalable means of remotely storing the quasi-continuous, fine grained outputs that give ES based applications a very light data footprint on a processing resource, and ensure negligible losses should the resource suddenly vanish. We will describe the motivations for the ES system, its unique features and capabilities, its architecture and the highly scalable tools and technologies employed in its implementation, and its applications in ATLAS processing on HPCs, commercial cloud resources, volunteer computing, and grid resources. Notice: This manuscript has been authored by employees of Brookhaven Science Associates, LLC under Contract No. DE-AC02-98CH10886 with the U.S. Department of Energy. The publisher by accepting the manuscript for publication acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.