FTS3: Quantitative Monitoring

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Riahi, H.^1 ; Salichos, M.^1 ; Keeble, O.^1 ; Andreeva, J.^1 ; Ayllon, A.A.^1 ; Di Girolamo, A.^1 ; Magini, N.^2 ; Roiser, S.^1 ; Simon, M.K.^1

European Organization for Nuclear Research, IT Department, Geneva

CH-1211- 23, Switzerland^1

Fermi National Laboratory, Batavia

IL

60510, United States^2

关键词: Data distribution;    LHC computing grids;    Monitoring information;    Network resource;    Quantitative monitoring;    Shared network links;    Transfer efficiency;    Virtual organization;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/6/062051/pdf
DOI  :  10.1088/1742-6596/664/6/062051

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The overall success of LHC data processing depends heavily on stable, reliable and fast data distribution. The Worldwide LHC Computing Grid (WLCG) relies on the File Transfer Service (FTS) as the data movement middleware for moving sets of files from one site to another. This paper describes the components of FTS3 monitoring infrastructure and how they are built to satisfy the common and particular requirements of the LHC experiments. We show how the system provides a complete and detailed cross-virtual organization (VO) picture of transfers for sites, operators and VOs. This information has proven critical due to the shared nature of the infrastructure, allowing a complete view of all transfers on shared network links between various workflows and VOs using the same FTS transfer manager. We also report on the performance of the FTS service itself, using data generated by the aforementioned monitoring infrastructure both during the commissioning and the first phase of production. We also explain how this monitoring information and network metrics produced can be used both as a starting point for troubleshooting data transfer issues, but also as a mechanism to collect information such as transfer efficiency between sites, achieved throughput and its evolution over time, most common errors, etc, and take decision upon them to further optimize transfer workflows. The service setup is subject to sites policies to control the network resource usage, as well as all the VOs making use of the Grid resources at the site to satisfy their requirements. FTS3 is the new version of FTS and has been deployed in production in August 2014.