Scaling up ATLAS Event Service to production levels on opportunistic computing platforms

17th International Workshop on Advanced Computing and Analysis Techniques in Physics Research

物理学;计算机科学

Benjamin, D.^1 ; Caballero, J.^2 ; Ernst, M.^2 ; Guan, W.^3 ; Hover, J.^2 ; Lesny, D.^4 ; Maeno, T.^2 ; Nilsson, P.^2 ; Tsulaia, V.^5 ; Van Gemmeren, P.^6 ; Vaniachine, A.^7 ; Wang, F.^3 ; Wenaus, T.^2

Duke University, Durham

NC

27708, United States^1

Brookhaven National Laboratory, Upton

NY

11973, United States^2

University of Wisconsin, Madison

WI

53706, United States^3

University of Illinois at Urbana-Champaign, Champaign

IL

61801, United States^4

Lawrence Berkeley National Laboratory, Berkeley

CA

94720, United States^5

Argonne National Laboratory, 9700 South Cass Avenue, Argonne

IL

60439, United States^6

Tomsk State University, Lenina Avenue 36, Tomsk

634050, Russia^7

关键词: Event service;    High-level triggers;    Job scheduling;    Open science grid;    Opportunistic computing;    Production level;    Resource usage;    Spot instances;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/762/1/012027/pdf
DOI  :  10.1088/1742-6596/762/1/012027

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Continued growth in public cloud and HPC resources is on track to exceed the dedicated resources available for ATLAS on the WLCG. Examples of such platforms are Amazon AWS EC2 Spot Instances, Edison Cray XC30 supercomputer, backfill at Tier 2 and Tier 3 sites, opportunistic resources at the Open Science Grid (OSG), and ATLAS High Level Trigger farm between the data taking periods. Because of specific aspects of opportunistic resources such as preemptive job scheduling and data I/O, their efficient usage requires workflow innovations provided by the ATLAS Event Service. Thanks to the finer granularity of the Event Service data processing workflow, the opportunistic resources are used more efficiently. We report on our progress in scaling opportunistic resource usage to double-digit levels in ATLAS production.