Reliability Engineering Analysis of ATLAS Data Reprocessing Campaigns

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Vaniachine, A.^1 ; Golubkov, D.^2 ; Karpenko, D.^3

High Energy Physics Division, Argonne National Laboratory, 9700 South Cass Avenue, Argonne

IL

60439, United States^1

Experimental Physics Department, Institute for High Energy Physics, Protvino

142281, Russia^2

Department of Physics, University of Oslo, P.B. 1048, Blindern, Oslo

0316, Norway^3

关键词: Continuous improvements;    Cost monitoring;    Fault tolerance mechanisms;    Performance prediction;    Peta-scale;    Reliability engineering;    Scale-up;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032101/pdf
DOI  :  10.1088/1742-6596/513/3/032101

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

During three years of LHC data taking, the ATLAS collaboration completed three petascale data reprocessing campaigns on the Grid, with up to 2 PB of data being reprocessed every year. In reprocessing on the Grid, failures can occur for a variety of reasons, while Grid heterogeneity makes failures hard to diagnose and repair quickly. As a result, Big Data processing on the Grid must tolerate a continuous stream of failures, errors and faults. While ATLAS fault-tolerance mechanisms improve the reliability of Big Data processing in the Grid, their benefits come at costs and result in delays making the performance prediction difficult. Reliability Engineering provides a framework for fundamental understanding of the Big Data processing on the Grid, which is not a desirable enhancement but a necessary requirement. In ATLAS, cost monitoring and performance prediction became critical for the success of the reprocessing campaigns conducted in preparation for the major physics conferences. In addition, our Reliability Engineering approach supported continuous improvements in data reprocessing throughput during LHC data taking. The throughput doubled in 2011 vs. 2010 reprocessing, then quadrupled in 2012 vs. 2011 reprocessing. We present the Reliability Engineering analysis of ATLAS data reprocessing campaigns providing the foundation needed to scale up the Big Data processing technologies beyond the petascale.