Archiving tools for EOS

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Sindrilaru, Elvin-Alin^1 ; Peters, Andreas-Joachim^1 ; Duellmann, Dirk^1

CERN IT, Geneva, Switzerland^1

关键词: Aggregate throughput;    Client interface;    Critical operations;    LHC experiments;    Message passing paradigms;    On-line analysis;    Point-to-point communication;    Process-based approach;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/4/042049/pdf
DOI  :  10.1088/1742-6596/664/4/042049

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Archiving data to tape is a critical operation for any storage system, especially for the EOS system at CERN which holds production data for all major LHC experiments. Each collaboration has an allocated quota it can use at any given time therefore, a mechanism for archiving "stale" data is needed so that storage space is reclaimed for online analysis operations. The archiving tool that we propose for EOS aims to provide a robust client interface for moving data between EOS and CASTOR (tape backed storage system) while enforcing best practices when it comes to data integrity and verification. All data transfers are done using a third-party copy mechanism which ensures point-to- point communication between the source and destination, thus providing maximum aggregate throughput. Using ZMQ message-passing paradigm and a process-based approach enabled us to achieve optimal utilisation of the resources and a stateless architecture which can easily be tuned during operation. The modular design and the implementation done in a high-level language like Python, has enabled us to easily extended the code base to address new demands like offering full and incremental backup capabilities.