Estimating job runtime for CMS analysis jobs

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Sfiligoi, I.^1

University of California San Diego, 9500 Gilman Dr, San Diego

CA

92093, United States^1

关键词: Compact Muon solenoids;    High confidence;    Historical data;    Leased resources;    Pilot system;    Runtimes;    Scheduling systems;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032087/pdf
DOI  :  10.1088/1742-6596/513/3/032087

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The basic premise of pilot systems is to create an overlay scheduling system on top of leased resources. And by definition, leases have a limited lifetime, so any job that is scheduled on such resources must finish before the lease is over, or it will be killed and all the computation is wasted. In order to effectively schedule jobs to resources, the pilot system thus requires the expected runtime of the users' jobs. Past studies have shown that relying on user provided estimates is not a valid strategy, so the system should try to make an estimate by itself. This paper provides a study of the historical data obtained from the Compact Muon Solenoid (CMS) experiment's Analysis Operations submission system. Clear patterns are observed, suggesting that making prediction of an expected job lifetime range is achievable with high confidence level in this environment.