Word2vec and dictionary based approach for uyghur text filtering

2nd Annual International Conference on Information System and Artificial Intelligence

物理学;计算机科学

Tohti, Turdi^1 ; Zhao, Yunxing^1 ; Musajan, Winira^1

School of Information Science and Engineering, Xinjiang University, Faculty of Computer, Science and Technology Building, No.666, Road, Shengli Urumqi

830046, China^1

关键词: Dictionary-based;    Filtering accuracies;    Similar pattern;    Text filtering;    Text information;    Uyghur-chinese;    Word vectors;    Wu-manber algorithms;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/887/1/012013/pdf
DOI  :  10.1088/1742-6596/887/1/012013

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

With emerging of deep learning, the expression of words in computer has made major breakthroughs and the effect of text processing based on word vector has also been significantly improved. This paper maps all patterns into a more abstract vector space by Uyghur-Chinese dictionary and deep learning tool Word2vec, at first. Secondly, a similar pattern is found according the characteristics of the original pattern. Finally, texts are filtered using Wu-Manber algorithm. Experiments show that this method can get obvious filtering accuracy and recall of Uyghur text information improved.