Large-Scale Merging of Histograms using Distributed In-Memory Computing

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Blomer, Jakob^1 ; Ganis, Gerardo^1

CERN, Geneva, Switzerland^1

关键词: Bisection bandwidth;    Distributed key-value stores;    Distributed merging;    Intermediate results;    Local networks;    Merging algorithms;    Running time;    Worker nodes;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/9/092003/pdf
DOI  :  10.1088/1742-6596/664/9/092003

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Most high-energy physics analysis jobs are embarrassingly parallel except for the final merging of the output objects, which are typically histograms. Currently, the merging of output histograms scales badly. The running time for distributed merging depends not only on the overall number of bins but also on the number partial histogram output files. That means, while the time to analyze data decreases linearly with the number of worker nodes, the time to merge the histograms in fact increases with the number of worker nodes. On the grid, merging jobs that take a few hours are not unusual. In order to improve the situation, we present a distributed and decentral merging algorithm whose running time is independent of the number of worker nodes. We exploit full bisection bandwidth of local networks and we keep all intermediate results in memory. We present benchmarks from an implementation using the parallel ROOT facility (PROOF) and RAMCloud, a distributed key-value store that keeps all data in DRAM.