Tier 3 batch system data locality via managed caches

16th International workshop on Advanced Computing and Analysis Techniques in physics research

物理学;计算机科学

Fischer, Max^1 ; Giffels, Manuel^1 ; Jung, Christopher^1 ; Kühn, Eileen^1 ; Quast, Günter^1

Karlsruhe Institute for Technology, Kaiserstraße 12, Karlsruhe

76131, Germany^1

关键词: Performance and scalabilities;    Performance data;    Resource usage;    Scheduling jobs;    Site boundary;    State information;    Storage hierarchy;    Storage systems;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/608/1/012018/pdf
DOI  :  10.1088/1742-6596/608/1/012018

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Modern data processing increasingly relies on data locality for performance and scalability, whereas the common HEP approaches aim for uniform resource pools with minimal locality, recently even across site boundaries. To combine advantages of both, the High- Performance Data Analysis (HPDA) Tier 3 concept opportunistically establishes data locality via coordinated caches. In accordance with HEP Tier 3 activities, the design incorporates two major assumptions: First, only a fraction of data is accessed regularly and thus the deciding factor for overall throughput. Second, data access may fallback to non-local, making permanent local data availability an inefficient resource usage strategy. Based on this, the HPDA design generically extends available storage hierarchies into the batch system. Using the batch system itself for scheduling file locality, an array of independent caches on the worker nodes is dynamically populated with high-profile data. Cache state information is exposed to the batch system both for managing caches and scheduling jobs. As a result, users directly work with a regular, adequately sized storage system. However, their automated batch processes are presented with local replications of data whenever possible.