A Novel Configuration Tuning Method Based on Feature Selection for Hadoop MapReduce

IEEE Access

Mingwei Lin¹  Guangxia Xu²  Sule Tang²  Chuang Ma²  Jun Liu² 

[1] College of Mathematics and Informatics, Fujian Normal University, Fuzhou, China;School of Software Engineering, Chongqing University of Posts and Telecommunications, Chongqing, China;

关键词: Parameter tuning;    Hadoop MapReduce;    kernel function;    K-means;   

DOI  :  10.1109/ACCESS.2020.2984778

来源: DOAJ

【 摘 要 】

Configuration parameter optimization is an important means of improving the performance of the MapReduce model. The existing parameter tuning methods usually optimize all configuration parameters in MapReduce. However, it is exceedingly challenging to tune all the parameters for the MapReduce model because there are massive configuration parameters in MapReduce. In this paper, a novel configuration parameter tuning method based on a feature selection algorithm is proposed, and it is composed of the feature selection objective function and feature selection process. The objective function is based on the kernel clustering algorithm, in which anisotropic Gaussian kernel is adopted instead of the traditional Gaussian kernel to accurately judge the importance of each parameter in MapReduce. Then, the relationship between the configuration parameters in MapReduce and the features in the feature selection algorithm is defined. Moreover, the importance of each parameter is reflected by the kernel width of anisotropic Gaussian kernels. At the same time, the method of gradient descent is introduced to update the kernel width and control the feature selection process of the iterative algorithm. Finally, experimental results show that the proposed algorithm performs suitably for the MapReduce model.

【 授权许可】

Unknown