Fuzzy Pool Balance: An algorithm to achieve a two dimensional balance in distribute storage systems

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Wu, Wenjing^1 ; Chen, Gang^1

IHEP, 19B Yuquan Road, Beijing

100049, China^1

关键词: Correction factors;    Distributed storage system;    File distribution;    Production system;    Resource utilizations;    Scheduling module;    Storage systems;    System throughput;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/4/042053/pdf
DOI  :  10.1088/1742-6596/513/4/042053

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The limitation of scheduling modules and the gradual addition of disk pools in distributed storage systems often result in imbalances among their disk pools in terms of both disk usage and file count. This can cause various problems to the storage system such as single point of failure, low system throughput and imbalanced resource utilization and system loads. An algorithm named Fuzzy Pool Balance (FPB) is proposed here to solve this problem. The input of FPB is the current file distribution among disk pools and the output is a file migration plan indicating what files are to be migrated to which pools. FPB uses an array to classify the files by their sizes. The file classification array is dynamically calculated with a defined threshold named Tmaxthat defines the allowed pool disk usage deviations. File classification is the basis of file migration. FPB also defines the Immigration Pool (IP) and Emigration Pool (EP) according to the pool disk usage and File Quantity Ratio (FQR) that indicates the percentage of each category of files in each disk pool, so files with higher FQR in an EP will be migrated to IP(s) with a lower FQR of this file category. To verify this algorithm, we implemented FPB on an ATLAS Tier2 dCache production system. The results show that FPB can achieve a very good balance in both free space and file counts, and adjusting the threshold value Tmaxand the correction factor to the average FQR can achieve a tradeoff between free space and file count.