An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers

Balman, Mehmet ; Kosar, Tevfik

Lawrence Berkeley National Laboratory

关键词: Storage;    Data Processing Advance Reservation, Resource Allocation, Scheduling, Data Management, Distributed Computing;    Processing;    Data Acquisition;    Data Base Management;   

DOI  :  10.2172/1050437
RP-ID  :  LBNL-5023E
RP-ID  :  DE-AC02-05CH11231
RP-ID  :  1050437

美国|英语

来源: UNT Digital Library

【 摘 要 】

Scientific applications and experimental facilities generate massive data sets that need to be transferred to remote collaborating sites for sharing, processing, and long term storage. In order to support increasingly data-intensive science, next generation research networks have been deployed to provide high-speed on-demand data access between collaborating institutions. In this paper, we present a practical model for online data scheduling in which data movement operations are scheduled in advance for end-to-end high performance transfers. In our model, data scheduler interacts with reservation managers and data transfer nodes in order to reserve available bandwidth to guarantee completion of jobs that are accepted and confirmed to satisfy preferred time constraint given by the user. Our methodology improves current systems by allowing researchers and higher level meta-schedulers to use data placement as a service where theycan plan ahead and reserve the scheduler time in advance for their data movement operations. We have implemented our algorithm and examined possible techniques for incorporation into current reservation frameworks. Performance measurements confirm that the proposed algorithm is efficient and scalable.