Research on MapReduce Task Scheduling Optimization

2018 2nd annual International Conference on Cloud Technology and Communication Engineering

计算机科学;无线电电子学

Ren, Ying^1 ; Li, Huawei^2 ; Wang, Lina^1

Naval Aeronautical University, Shandong, China^1

Shan Dong Businss Institute, Yantai, Shandong

264001, China^2

关键词: Job management;    Map-reduce;    Resource consumption;    Resource optimization;    Scheduling strategies;    Task-scheduling;    Time allocation;    Time requirements;   

Others  :  https://iopscience.iop.org/article/10.1088/1757-899X/466/1/012016/pdf
DOI  :  10.1088/1757-899X/466/1/012016

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

One of the big challenges in MapReduce is how to effectively allocate resources for the jobs submitted by users , and finish the job within the time specified by the user .The original scheduling strategy lacks the ability of job management and can not complete the job according to the time requirement of the user. In this paper, a task scheduler (ET-scheduler) is proposed to meet the needs of job time and resource optimization. It can not only meet the time needs of users, but also minimize the resource consumption and adjust the time allocation in the process of map and reduce. Experiments show that the algorithm not only completes the most jobs in a given time, but also minimizes the resource consumption in the Hadoop cluster.