Cloud Bursting with GlideinWMS: Means to satisfy ever increasing computing needs for Scientific Workflows

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Mhashilkar, Parag^1 ; Tiradani, Anthony^1 ; Holzman, Burt^1 ; Larson, Krista^1 ; Sfiligoi, Igor^2 ; Rynge, Mats^3

Fermi National Accelerator Laboratory, P O Box 500, Batavia

IL-60510, United States^1

University of California at San Diego, Department of Physics, 9500 Gilman Drive, San Diego

CA

92093-0354, United States^2

Information Sciences Institute (ISI), 4676 Admiralty Way, Marina del Rey

CA

90292, United States^3

关键词: Architectural changes;    Cloud infrastructures;    Complex applications;    Computational resources;    Grid infrastructures;    Scientific community;    Scientific workflows;    Workload management;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032069/pdf
DOI  :  10.1088/1742-6596/513/3/032069

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

Scientific communities have been in the forefront of adopting new technologies and methodologies in the computing. Scientific computing has influenced how science is done today, achieving breakthroughs that were impossible to achieve several decades ago. For the past decade several such communities in the Open Science Grid (OSG) and the European Grid Infrastructure (EGI) have been using GlideinWMS to run complex application workflows to effectively share computational resources over the grid. GlideinWMS is a pilot-based workload management system (WMS) that creates on demand, a dynamically sized overlay HTCondor batch system on grid resources. At present, the computational resources shared over the grid are just adequate to sustain the computing needs. We envision that the complexity of the science driven by Big Data will further push the need for computational resources. To fulfill their increasing demands and/or to run specialized workflows, some of the big communities like CMS are investigating the use of cloud computing as Infrastructure-As-A-Service (IAAS) with GlideinWMS as a potential alternative to fill the void. Similarly, communities with no previous access to computing resources can use GlideinWMS to setup up a batch system on the cloud infrastructure. To enable this, the architecture of GlideinWMS has been extended to enable support for interfacing GlideinWMS with different Scientific and commercial cloud providers like HLT, FutureGrid, FermiCloud and Amazon EC2. In this paper, we describe a solution for cloud bursting with GlideinWMS. The paper describes the approach, architectural changes and lessons learned while enabling support for cloud infrastructures in GlideinWMS.