Operational experience running Hadoop XRootD Fallback

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Dost, J.M.^1 ; Tadel, A.^1 ; Tadel, M.^1 ; Würthwein, F.^1

University of California, 9500 Gilman Dr, San Diego

CA

92093, United States^1

关键词: Data redundancy;    File replication;    File systems;    Operational experience;    Software systems;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/4/042011/pdf
DOI  :  10.1088/1742-6596/664/4/042011

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

In April of 2014, the UCSD T2 Center deployed hdfs-xrootd-fallback, a UCSD- developed software system that interfaces Hadoop with XRootD to increase reliability of the Hadoop file system. The hdfs-xrootd-fallback system allows a site to depend less on local file replication and more on global replication provided by the XRootD federation to ensure data redundancy. Deploying the software has allowed us to reduce Hadoop replication on a significant subset of files in our cluster, freeing hundreds of terabytes in our local storage, and to recover HDFS blocks lost due to storage degradation. An overview of the architecture of the hdfs-xrootd-fallback system will be presented, as well as details of our experience operating the service over the past year.