A History-based Estimation for LHCb job requirements

21st International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Rauschmayr, Nathalie^1

CERN - European Organization for Nuclear Research, Switzerland^1

关键词: Experiment condition;    Memory consumption;    Memory footprint;    Production manager;    Software versions;    Utilization of resources;    Workload analysis;    Workload management;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/664/6/062050/pdf
DOI  :  10.1088/1742-6596/664/6/062050

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The main goal of a Workload Management System (WMS) is to find and allocate resources for the given tasks. The more and better job information the WMS receives, the easier will be to accomplish its task, which directly translates into higher utilization of resources. Traditionally, the information associated with each job, like expected runtime, is defined beforehand by the Production Manager in best case and fixed arbitrary values by default. In the case of LHCb's Workload Management System no mechanisms are provided which automate the estimation of job requirements. As a result, much more CPU time is normally requested than actually needed. Particularly, in the context of multicore jobs this presents a major problem, since single- and multicore jobs shall share the same resources. Consequently, grid sites need to rely on estimations given by the VOs in order to not decrease the utilization of their worker nodes when making multicore job slots available. The main reason for going to multicore jobs is the reduction of the overall memory footprint. Therefore, it also needs to be studied how memory consumption of jobs can be estimated. A detailed workload analysis of past LHCb jobs is presented. It includes a study of job features and their correlation with runtime and memory consumption. Following the features, a supervised learning algorithm is developed based on a history based prediction. The aim is to learn over time how jobs' runtime and memory evolve influenced due to changes in experiment conditions and software versions. It will be shown that estimation can be notably improved if experiment conditions are taken into account.