How much higher can HTCondor fly?

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Fajardo, E.M.^1 ; Dost, J.M.^1 ; Holzman, B.^2 ; Tannenbaum, T.^3 ; Letts, J.^1 ; Tiradani, A.^2 ; Bockelman, B.^4 ; Frey, J.^3 ; Mason, D.^2

University of California San Diego, San Diego

CA, United States^1

Fermi National Accelerator Laboratory, United States^2

University of Wisconsin, Madison

WI, United States^3

University of Nebraska, Lincoln

NE, United States^4

关键词: Compact Muon solenoids;    Configuration options;    Distributed dynamics;    High-throughput computing;    LHC computing grids;    Scaling results;    Sustained operations;    Test condition;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/6/062014/pdf
DOI  :  10.1088/1742-6596/664/6/062014

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The HTCondor high throughput computing system is heavily used in the high energy physics (HEP) community as the batch system for several Worldwide LHC Computing Grid (WLCG) resources. Moreover, it is the backbone of GlidelnWMS, the pilot system used by the computing organization of the Compact Muon Solenoid (CMS) experiment. To prepare for LHC Run 2, we probed the scalability limits of new versions and configurations of HTCondor with a goal of reaching 200,000 simultaneous running jobs in a single internationally distributed dynamic pool. In this paper, we first describe how we created an opportunistic distributed testbed capable of exercising runs with 200,000 simultaneous jobs without impacting production. This testbed methodology is appropriate not only for scale testing HTCondor, but potentially for many other services. In addition to the test conditions and the testbed topology, we include the suggested configuration options used to obtain the scaling results, and describe some of the changes to HTCondor inspired by our testing that enabled sustained operations at scales well beyond previous limits.