Mining on Big Data Using Hadoop MapReduce Model

14th International Conference on Science, Engineering and Technology

自然科学;工业技术

Salman Ahmed, G.^1 ; Bhattacharya, Sweta^1

VIT University, Vellore

Tamil Nadu

632014, India^1

关键词: Hadoop MapReduce;    Locality sensitive hashing;    Map-reduce programming;    Market basket;    Parallel calculation;    Set mining;    Similar datum;   

Others  :  https://iopscience.iop.org/article/10.1088/1757-899X/263/4/042007/pdf
DOI  :  10.1088/1757-899X/263/4/042007

来源: IOP

【 摘 要 】

Customary parallel calculations for mining nonstop item create opportunity to adjust stack of similar data among hubs. The paper aims to review this process by analyzing the critical execution downside of the common parallel recurrent item-set mining calculations. Given a larger than average dataset, data apportioning strategies inside the current arrangements endure high correspondence and mining overhead evoked by repetitive exchanges transmitted among registering hubs. We tend to address this downside by building up a learning apportioning approach referred as Hadoop abuse using the map-reduce programming model. All objectives of Hadoop are to zest up the execution of parallel recurrent item-set mining on Hadoop bunches. Fusing the comparability metric and furthermore the locality-sensitive hashing procedure, Hadoop puts to a great degree comparative exchanges into an information segment to lift neighborhood while not making AN exorbitant assortment of excess exchanges. We tend to execute Hadoop on a 34-hub Hadoop bunch, driven by a decent change of datasets made by IBM quest market-basket manufactured data generator. Trial uncovers the fact that Hadoop contributes towards lessening system and processing masses by the uprightness of dispensing with excess exchanges on Hadoop hubs. Hadoop impressively outperforms and enhances the other models considerably.