BESIU Physical Analysis on Hadoop Platform

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Huo, Jing^1,2 ; Zang, Dongsong^1,2 ; Lei, Xiaofeng^1,2 ; Li, Qiang^1,2 ; Sun, Gongxing^1

Institute of High Energy Physics, Beijing, China^1

University of Chinese Academy of Sciences, Beijing, China^2

关键词: Computing clusters;    Data analysis system;    Distributed computing technology;    High concurrencies;    High fault tolerances;    High scalabilities;    Map-reduce programming;    Physical analysis;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032044/pdf
DOI  :  10.1088/1742-6596/513/3/032044

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

In the past 20 years, computing cluster has been widely used for High Energy Physics data processing. The jobs running on the traditional cluster with a Data-to-Computing structure, have to read large volumes of data via the network to the computing nodes for analysis, thereby making the I/O latency become a bottleneck of the whole system. The new distributed computing technology based on the MapReduce programming model has many advantages, such as high concurrency, high scalability and high fault tolerance, and it can benefit us in dealing with Big Data. This paper brings the idea of using MapReduce model to do BESIII physical analysis, and presents a new data analysis system structure based on Hadoop platform, which not only greatly improve the efficiency of data analysis, but also reduces the cost of system building. Moreover, this paper establishes an event pre-selection system based on the event level metadata(TAGs) database to optimize the data analyzing procedure.