Exploiting volatile opportunistic computing resources with Lobster

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Woodard, Anna^1 ; Wolf, Matthias^1 ; Mueller, Charles^1 ; Tovar, Ben^1 ; Donnelly, Patrick^1 ; Anampa, Kenyi Hurtado^1 ; Brenner, Paul^1 ; Lannon, Kevin^1 ; Hildreth, Mike^1 ; Thain, Douglas^1

University of Notre Dame, Notre Dame

IN

46556, United States^1

关键词: Compact Muon solenoids;    Computer scientists;    High energy physics experiments;    Large Hadron collider LHC;    Monitoring and diagnostics;    Opportunistic computing;    Software environments;    Workflow managements;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/3/032035/pdf
DOI  :  10.1088/1742-6596/664/3/032035

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Analysis of high energy physics experiments using the Compact Muon Solenoid (CMS) at the Large Hadron Collider (LHC) can be limited by availability of computing resources. As a joint effort involving computer scientists and CMS physicists at Notre Dame, we have developed an opportunistic workflow management tool, Lobster, to harvest available cycles from university campus computing pools. Lobster consists of a management server, file server, and worker processes which can be submitted to any available computing resource without requiring root access. Lobster makes use of the Work Queue system to perform task management, while the CMS specific software environment is provided via CVMFS and Parrot. Data is handled via Chirp and Hadoop for local data storage and XrootD for access to the CMS wide-area data federation. An extensive set of monitoring and diagnostic tools have been developed to facilitate system optimisation. We have tested Lobster using the 20 000-core cluster at Notre Dame, achieving approximately 8-10k tasks running simultaneously, sustaining approximately 9 Gbit/s of input data and 340 Mbit/s of output data.