Research on Spark Big Data Recommendation Algorithm under Hadoop Platform

2018 4th International Conference on Environmental Science and Material Application

生态环境科学;材料科学

Huang, Zubang^1

East China University of Technology, Nanchang

330000, China^1

关键词: Big data applications;    Data processing methods;    Hadoop platforms;    Online learning;    Processing capability;    Recommendation algorithms;    Speed requirement;    Task-scheduling;   

Others  :  https://iopscience.iop.org/article/10.1088/1755-1315/252/3/032103/pdf
DOI  :  10.1088/1755-1315/252/3/032103

来源: IOP

【 摘 要 】

Hadoop is the main platform for big data mining. It can build Yarn, Mahout, Storm, Graph Lab and other frameworks on its platform. Spark framework is a common task scheduling framework, which not only has powerful processing capabilities, but also has Data processing timeliness, which integrates functions such as machine learning, graph calculation, and online learning, adopts a unified processing mechanism, and the speed is dozens or even hundreds of times that of traditional data processing methods. The paper establishes a Spark framework based on the Hadoop platform, and verifies that the Spark framework's processing mechanism for big data can meet the speed requirements of the increasingly value-added big data applications.