XRootd, disk-based, caching proxy for optimization of data access, data placement and data replication

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Bauerdick, L.A.T.^1 ; Bloom, K.^3 ; Bockelman, B.^3 ; Bradley, D.C.^4 ; Dasu, S.^4 ; Dost, J.M.^2 ; Sfiligoi, I.^2 ; Tadel, A.^2 ; Tadel, M.^2 ; Wuerthwein, F.^2 ; Yagil, A.^2

Fermilab, Batavia

IL

60510-5011, United States^1

UC San Diego, San Diego

CA

92093, United States^2

University of Nebraska-Lincoln, Lincoln

NE

68588, United States^3

University of Wisconsin-Madison, Madison

WI

53706, United States^4

关键词: Data federation;    Data placement;    Data replication;    File access;    Hadoop distributed file systems;    Implementation support;    Network access;    Pre-production;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/4/042044/pdf
DOI  :  10.1088/1742-6596/513/4/042044

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Following the success of the XRootd-based US CMS data federation, the AAA project investigated extensions of the federation architecture by developing two sample implementations of an XRootd, disk-based, caching proxy. The first one simply starts fetching a whole file as soon as a file open request is received and is suitable when completely random file access is expected or it is already known that a whole file be read. The second implementation supports on-demand downloading of partial files. Extensions to the Hadoop Distributed File System have been developed to allow for an immediate fallback to network access when local HDFS storage fails to provide the requested block. Both cache implementations are in pre-production testing at UCSD.