Single-Pass Clustering Algorithm Based on Storm

2017 International Conference on Control Engineering and Artificial Intelligence

计算机科学

Fang, L.I.^1 ; Longlong, D.A.I.^1 ; Zhiying, Jiang^1 ; Shunzi, L.I.^1

Beijing University of Chemical Technology, Beijing

100029, China^1

关键词: Clustering accuracy;    Parallel clustering;    Real time performance;    Single-pass algorithm;    Single-pass clustering;    Stream data clustering;    Topic detection and tracking;    Traditional clustering;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/806/1/012017/pdf
DOI  :  10.1088/1742-6596/806/1/012017

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

The dramatically increasing volume of data makes the computational complexity of traditional clustering algorithm rise rapidly accordingly, which leads to the longer time. So as to improve the efficiency of the stream data clustering, a distributed real-time clustering algorithm (S-Single-Pass) based on the classic Single-Pass [1] algorithm and Storm [2] computation framework was designed in this paper. By employing this kind of method in the Topic Detection and Tracking (TDT) [3], the real-time performance of topic detection arises effectively. The proposed method splits the clustering process into two parts: one part is to form clusters for the multi-thread parallel clustering, the other part is to merge the generated clusters in the previous process and update the global clusters. Through the experimental results, the conclusion can be drawn that the proposed method have the nearly same clustering accuracy as the traditional Single-Pass algorithm and the clustering accuracy remains steady, computing rate increases linearly when increasing the number of cluster machines and nodes (processing threads).