POSIX and Object Distributed Storage Systems Performance Comparison Studies With Real-Life Scenarios in an Experimental Data Taking Context Leveraging OpenStack Swift & Ceph

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Poat, M.D.^1 ; Lauret, J.^1 ; Betts, W.^1

Brookhaven National Laboratory, P.O. Box 5000, Upton

NY

11973-5000, United States^1

关键词: Data collection;    Distributed storage system;    Limited storage;    Object storages;    Online computing;    Performance tests;    Real-world scenario;    True potentials;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/4/042031/pdf
DOI  :  10.1088/1742-6596/664/4/042031

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The STAR online computing infrastructure has become an intensive dynamic system used for first-hand data collection and analysis resulting in a dense collection of data output. As we have transitioned to our current state, inefficient, limited storage systems have become an impediment to fast feedback to online shift crews. Motivation for a centrally accessible, scalable and redundant distributed storage system had become a necessity in this environment. OpenStack Swift Object Storage and Ceph Object Storage are two eye-opening technologies as community use and development have led to success elsewhere. In this contribution, OpenStack Swift and Ceph have been put to the test with single and parallel I/O tests, emulating real world scenarios for data processing and workflows. The Ceph file system storage, offering a POSIX compliant file system mounted similarly to an NFS share was of particular interest as it aligned with our requirements and was retained as our solution. I/O performance tests were run against the Ceph POSIX file system and have presented surprising results indicating true potential for fast I/O and reliability. STAR'S online compute farm historical use has been for job submission and first hand data analysis. The goal of reusing the online compute farm to maintain a storage cluster and job submission will be an efficient use of the current infrastructure.