MEST: A Model-Driven Efficient Searching Approach for MapReduce Self-Tuning

IEEE Access

Zhibin Yu¹  Qixiao Liu¹  Zhendong Bei¹  Shengzhong Feng¹  Chengzhong Xu¹  Shuang Song² 

[1] Chinese Academy of Sciences, Shenzhen Institutes of Advanced Technology, Shenzhen, China;The University of Texas at Austin, Austin, TX, USA;

关键词: MapReduce;    Hadoop;    self-tuning;    model trees;    genetic algorithm;   

DOI  :  10.1109/ACCESS.2017.2672675

来源: DOAJ

【 摘 要 】

Hadoop is the most popular implementation framework of the MapReduce programming model, and it has a number of performance-critical configuration parameters. However, manually setting these parameters to their optimal values not only needs in-depth knowledge on Hadoop as well as the job itself, but also requires a large amount of time and efforts. Automatic approaches have therefore been proposed. Their usage, however, is still quite limited due to the intolerably long searching time. In this paper, we introduce MapreducE Self-Tuning (MEST), a framework that accelerates the searching process for the optimal configuration of a given Hadoop application. We have devised a novel mechanism by integrating the model trees algorithm with the genetic algorithm. As such, MEST significantly reduces the searching time by removing unnecessary profiling, modeling, and searching steps, which are mandatory for existing approaches. Our experiments using five benchmarks, each with two input data sets (DS1 and 2× DS1 ) show that MEST improves the searching efficiency (SE) by factors of 1.37× and 2.18×$ on average respectively over the state-of-the-art approach.

【 授权许可】

Unknown